Description

Background and Context

Businesses like banks that provide service have to worry about the problem of 'Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on the improvement of service, keeping in mind these priorities.

Objective

Given a Bank customer, build a neural network-based classifier that can determine whether they will leave or not in the next 6 months.

Data Description

The case study is from an open-source dataset from Kaggle. The dataset contains 10,000 sample points with 14 distinct features such as CustomerId, CreditScore, Geography, Gender, Age, Tenure, Balance, etc.

Data Dictionary

• CustomerId: Unique ID which is assigned to each customer
• Surname: Last name of the customer
• CreditScore: It defines the credit history of the customer.
• Geography: A customer’s location
• Gender: It defines the Gender of the customer
• Age: Age of the customer
• Tenure: Number of years for which the customer has been with the bank
• NumOfProducts: It refers to the number of products that a customer has purchased through the bank.
• Balance: Account balance
• HasCrCard: It is a categorical variable that decides whether the customer has a credit card or not.
• EstimatedSalary: Estimated salary
• isActiveMember: It is a categorical variable that decides whether the customer is an active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions, etc )
• Excited: It is a categorical variable that decides whether the customer left the bank within six months or not. It can take two values

                * 0=No ( Customer did not leave the bank )
                * 1=Yes ( Customer left the bank )

Reading Dataset and Feature Elimination

• Read the dataset properly - Print the overview of the data (statistical summary, shape, info, etc) - Eliminate the unique features from the dataset with proper reasoning

import tensorflow as tf
print(tf.__version__)

2.8.0

import pandas as pd
import numpy as np
import seaborn as sns
import sklearn     
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
import matplotlib.pyplot as plt

churn_data = pd.read_csv('churn.csv')
churn_data

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	1	15634602	Hargrave	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	2	15647311	Hill	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	3	15619304	Onio	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	4	15701354	Boni	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	5	15737888	Mitchell	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
9995	9996	15606229	Obijiaku	771	France	Male	39	5	0.00	2	1	0	96270.64	0
9996	9997	15569892	Johnstone	516	France	Male	35	10	57369.61	1	1	1	101699.77	0
9997	9998	15584532	Liu	709	France	Female	36	7	0.00	1	0	1	42085.58	1
9998	9999	15682355	Sabbatini	772	Germany	Male	42	3	75075.31	2	1	0	92888.52	1
9999	10000	15628319	Walker	792	France	Female	28	4	130142.79	1	1	0	38190.78	0

10000 rows × 14 columns

churn_data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 14 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   RowNumber        10000 non-null  int64  
 1   CustomerId       10000 non-null  int64  
 2   Surname          10000 non-null  object 
 3   CreditScore      10000 non-null  int64  
 4   Geography        10000 non-null  object 
 5   Gender           10000 non-null  object 
 6   Age              10000 non-null  int64  
 7   Tenure           10000 non-null  int64  
 8   Balance          10000 non-null  float64
 9   NumOfProducts    10000 non-null  int64  
 10  HasCrCard        10000 non-null  int64  
 11  IsActiveMember   10000 non-null  int64  
 12  EstimatedSalary  10000 non-null  float64
 13  Exited           10000 non-null  int64  
dtypes: float64(2), int64(9), object(3)
memory usage: 1.1+ MB

# Let's check for duplicate values in the data
churn_data.duplicated().sum()

0

# Let's check for missing values in the data
round(churn_data.isnull().sum() / churn_data.isnull().count() * 100, 2)

RowNumber          0.0
CustomerId         0.0
Surname            0.0
CreditScore        0.0
Geography          0.0
Gender             0.0
Age                0.0
Tenure             0.0
Balance            0.0
NumOfProducts      0.0
HasCrCard          0.0
IsActiveMember     0.0
EstimatedSalary    0.0
Exited             0.0
dtype: float64

churn_data.describe().T

	count	mean	std	min	25%	50%	75%	max
RowNumber	10000.0	5.000500e+03	2886.895680	1.00	2500.75	5.000500e+03	7.500250e+03	10000.00
CustomerId	10000.0	1.569094e+07	71936.186123	15565701.00	15628528.25	1.569074e+07	1.575323e+07	15815690.00
CreditScore	10000.0	6.505288e+02	96.653299	350.00	584.00	6.520000e+02	7.180000e+02	850.00
Age	10000.0	3.892180e+01	10.487806	18.00	32.00	3.700000e+01	4.400000e+01	92.00
Tenure	10000.0	5.012800e+00	2.892174	0.00	3.00	5.000000e+00	7.000000e+00	10.00
Balance	10000.0	7.648589e+04	62397.405202	0.00	0.00	9.719854e+04	1.276442e+05	250898.09
NumOfProducts	10000.0	1.530200e+00	0.581654	1.00	1.00	1.000000e+00	2.000000e+00	4.00
HasCrCard	10000.0	7.055000e-01	0.455840	0.00	0.00	1.000000e+00	1.000000e+00	1.00
IsActiveMember	10000.0	5.151000e-01	0.499797	0.00	0.00	1.000000e+00	1.000000e+00	1.00
EstimatedSalary	10000.0	1.000902e+05	57510.492818	11.58	51002.11	1.001939e+05	1.493882e+05	199992.48
Exited	10000.0	2.037000e-01	0.402769	0.00	0.00	0.000000e+00	0.000000e+00	1.00

churn_data.shape

(10000, 14)

churn_data.isna().sum()

RowNumber          0
CustomerId         0
Surname            0
CreditScore        0
Geography          0
Gender             0
Age                0
Tenure             0
Balance            0
NumOfProducts      0
HasCrCard          0
IsActiveMember     0
EstimatedSalary    0
Exited             0
dtype: int64

churn_data.nunique()

RowNumber          10000
CustomerId         10000
Surname             2932
CreditScore          460
Geography              3
Gender                 2
Age                   70
Tenure                11
Balance             6382
NumOfProducts          4
HasCrCard              2
IsActiveMember         2
EstimatedSalary     9999
Exited                 2
dtype: int64

for i in churn_data.describe(include=["object"]).columns:
    print("Unique values in", i, "are :")
    print(churn_data[i].value_counts())
    print("*" * 50)

Unique values in Surname are :
Smith       32
Scott       29
Martin      29
Walker      28
Brown       26
            ..
Izmailov     1
Bold         1
Bonham       1
Poninski     1
Burbidge     1
Name: Surname, Length: 2932, dtype: int64
**************************************************
Unique values in Geography are :
France     5014
Germany    2509
Spain      2477
Name: Geography, dtype: int64
**************************************************
Unique values in Gender are :
Male      5457
Female    4543
Name: Gender, dtype: int64
**************************************************

df=churn_data.copy()
df=df.drop(["RowNumber","Surname"],axis=1)

churn_data.nunique()

RowNumber          10000
CustomerId         10000
Surname             2932
CreditScore          460
Geography              3
Gender                 2
Age                   70
Tenure                11
Balance             6382
NumOfProducts          4
HasCrCard              2
IsActiveMember         2
EstimatedSalary     9999
Exited                 2
dtype: int64

df.shape

(10000, 12)

Perform an Exploratory Data Analysis on the data

• Checked whether the dataset is balanced or not - Bivariate analysis - Use appropriate visualizations to identify the patterns and insights - Any other exploratory deep dive

Univariate Analysis

# function to plot a boxplot and a histogram along the same scale.


def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (12,7))
    kde: whether to show the density curve (default False)
    bins: number of bins for histogram (default None)
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,  # Number of rows of the subplot grid= 2
        sharex=True,  # x-axis will be shared among all subplots
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  # creating the 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  # boxplot will be created and a star will indicate the mean value of the column
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )  # For histogram
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    )  # Add mean to the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Add median to the histogram

df.columns

Index(['CustomerId', 'CreditScore', 'Geography', 'Gender', 'Age', 'Tenure',
       'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember',
       'EstimatedSalary', 'Exited'],
      dtype='object')

histogram_boxplot(df, "CreditScore")

histogram_boxplot(df, "Age")

histogram_boxplot(df, "Tenure")

histogram_boxplot(df, "Balance")

histogram_boxplot(df, "NumOfProducts")

histogram_boxplot(df, "HasCrCard")

histogram_boxplot(df, "EstimatedSalary")

histogram_boxplot(df, "Exited")

print(df.Exited.value_counts())
labels = 'Churned', 'Not Churned'
#sizes = [ds.is_promoted[ds['is_promoted']==1].count(), ds.is_promoted[ds['is_promoted']==0].count()]
sizes = [df.Exited[df['Exited']==1].count(),df.Exited[df['Exited']==0].count()]
explode = (0, 0.1)
fig1, ax1 = plt.subplots(figsize=(10, 8))
ax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',
        shadow=True, startangle=90)
ax1.axis('equal')
plt.title("Churn", size = 20)
plt.show()

0    7963
1    2037
Name: Exited, dtype: int64

df.columns

Index(['CustomerId', 'CreditScore', 'Geography', 'Gender', 'Age', 'Tenure',
       'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember',
       'EstimatedSalary', 'Exited'],
      dtype='object')

Bivariate Analysis

### Function to plot distributions


def distribution_plot_wrt_target(data, predictor, target):

    fig, axs = plt.subplots(2, 2, figsize=(12, 10))

    target_uniq = data[target].unique()

    axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
    sns.histplot(
        data=data[data[target] == target_uniq[0]],
        x=predictor,
        kde=True,
        ax=axs[0, 0],
        color="teal",
    )

    axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
    sns.histplot(
        data=data[data[target] == target_uniq[1]],
        x=predictor,
        kde=True,
        ax=axs[0, 1],
        color="orange",
    )

    axs[1, 0].set_title("Boxplot w.r.t target")
    sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")

    axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
    sns.boxplot(
        data=data,
        x=target,
        y=predictor,
        ax=axs[1, 1],
        showfliers=False,
        palette="gist_rainbow",
    )

    plt.tight_layout()
    plt.show()

# function to create labeled barplots


def labeled_barplot(data, feature, perc=False, n=None):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))

    plt.xticks(rotation=90, fontsize=15)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n].sort_values(),
    )

    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            )  # percentage of each class of the category
        else:
            label = p.get_height()  # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height()  # height of the plot

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage

    plt.show()  # show the plot

distribution_plot_wrt_target(df, "Age", "Exited")

distribution_plot_wrt_target(df, "CreditScore", "Exited")

# function to plot stacked bar chart


def stacked_barplot(data, predictor, target):
    """
    Print the category counts and plot a stacked bar chart

    data: dataframe
    predictor: independent variable
    target: target variable
    """
    count = data[predictor].nunique()
    sorter = data[target].value_counts().index[-1]
    tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
        by=sorter, ascending=False
    )
    print(tab1)
    print("-" * 120)
    tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
        by=sorter, ascending=False
    )
    tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
    plt.legend(
        loc="lower left",
        frameon=False,
    )
    plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
    plt.show()

stacked_barplot(df, "Gender", "Exited")

Exited     0     1    All
Gender                   
All     7963  2037  10000
Female  3404  1139   4543
Male    4559   898   5457
------------------------------------------------------------------------------------------------------------------------

stacked_barplot(df, "Tenure", "Exited")

Exited     0     1    All
Tenure                   
All     7963  2037  10000
1        803   232   1035
3        796   213   1009
9        771   213    984
5        803   209   1012
4        786   203    989
2        847   201   1048
8        828   197   1025
6        771   196    967
7        851   177   1028
10       389   101    490
0        318    95    413
------------------------------------------------------------------------------------------------------------------------

stacked_barplot(df, "Geography", "Exited")

Exited        0     1    All
Geography                   
All        7963  2037  10000
Germany    1695   814   2509
France     4204   810   5014
Spain      2064   413   2477
------------------------------------------------------------------------------------------------------------------------

stacked_barplot(df, "Age", "Exited")

Exited     0     1    All
Age                      
All     7963  2037  10000
46       135    91    226
40       343    89    432
43       209    88    297
45       142    87    229
..       ...   ...    ...
79         4     0      4
78         5     0      5
77        10     0     10
76        11     0     11
75         9     0      9

[71 rows x 3 columns]
------------------------------------------------------------------------------------------------------------------------

df.nunique()

CustomerId         10000
CreditScore          460
Geography              3
Gender                 2
Age                   70
Tenure                11
Balance             6382
NumOfProducts          4
HasCrCard              2
IsActiveMember         2
EstimatedSalary     9999
Exited                 2
dtype: int64

# for column in df.columns:
labeled_barplot(df, "Age", perc=True)

df.columns

Index(['CustomerId', 'CreditScore', 'Geography', 'Gender', 'Age', 'Tenure',
       'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember',
       'EstimatedSalary', 'Exited'],
      dtype='object')

labeled_barplot(df, "NumOfProducts", perc=True)

labeled_barplot(df, "Tenure", perc=True)

labeled_barplot(df, "IsActiveMember", perc=True)

labeled_barplot(df, "HasCrCard", perc=True)

labeled_barplot(df, "Exited", perc=True)

labeled_barplot(df, "Geography", perc=True)

labeled_barplot(df, "Gender", perc=True)

Bivariate analysis

plt.figure(figsize=(15, 7))
sns.heatmap(
    df.corr(), annot=True, vmin=-1, vmax=1, fmt=".3f", cmap="Spectral"
)
plt.show()

sns.pairplot(df,hue = 'Exited',diag_kind = "kde",kind = "scatter",palette = "husl",height=5.5)
plt.show()

Illustrate the insights based on EDA

-Key meaningful observations from Bivariate analysis

• Data has shape of 10000 rows and 14 columns
• Row Number and Surname is doesnt have significance and can be dropped.
• There are no duplicates
• Most of the customers are between age 25 to 45 range
• Over 95% customers used 1-2 Number of products
• Almost half 51% customers are Active member and 70% have credit card and can be looked into.
• Overall 20.4% Customers Churned
• Over 50% customers come from France. Germany and Spain contribute around 25% each.
• There is a positive correlation between Age, Balance, Salary with Churn and negative corelation with Active member and Number of Products
• People between age 50 to 60 tend to churn more.
• People between with Number of Products greater than 2 tend to churn more as well.
• More % of people in Germany churned

Data Pre-processing

• Split the target variable and predictors - Split the data into train and test - Rescale the data

df['CustomerId'] = df['CustomerId'].astype('float32')
df['CreditScore'] = df['CreditScore'].astype('float32')
df['Age'] = df['Age'].astype('float32')
df['Tenure'] = df['Tenure'].astype('float32')
df['NumOfProducts'] = df['NumOfProducts'].astype('float32')
df['HasCrCard'] = df['HasCrCard'].astype('float32')
df['IsActiveMember'] = df['IsActiveMember'].astype('float32')
df['Exited'] = df['Exited'].astype('float32')


df['Geography'] = df['Geography'].astype('category')
df['Gender'] = df['Gender'].astype('category')

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 12 columns):
 #   Column           Non-Null Count  Dtype   
---  ------           --------------  -----   
 0   CustomerId       10000 non-null  float32 
 1   CreditScore      10000 non-null  float32 
 2   Geography        10000 non-null  category
 3   Gender           10000 non-null  category
 4   Age              10000 non-null  float32 
 5   Tenure           10000 non-null  float32 
 6   Balance          10000 non-null  float64 
 7   NumOfProducts    10000 non-null  float32 
 8   HasCrCard        10000 non-null  float32 
 9   IsActiveMember   10000 non-null  float32 
 10  EstimatedSalary  10000 non-null  float64 
 11  Exited           10000 non-null  float32 
dtypes: category(2), float32(8), float64(2)
memory usage: 488.7 KB

churn_process_df = pd.get_dummies(df,drop_first=True)

churn_process_df = churn_process_df.astype('float32')
churn_process_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 13 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   CustomerId         10000 non-null  float32
 1   CreditScore        10000 non-null  float32
 2   Age                10000 non-null  float32
 3   Tenure             10000 non-null  float32
 4   Balance            10000 non-null  float32
 5   NumOfProducts      10000 non-null  float32
 6   HasCrCard          10000 non-null  float32
 7   IsActiveMember     10000 non-null  float32
 8   EstimatedSalary    10000 non-null  float32
 9   Exited             10000 non-null  float32
 10  Geography_Germany  10000 non-null  float32
 11  Geography_Spain    10000 non-null  float32
 12  Gender_Male        10000 non-null  float32
dtypes: float32(13)
memory usage: 507.9 KB

Y = churn_process_df['Exited']

# churn_model_df=df.copy()
X=churn_process_df.drop("Exited",axis=1)

Splitting the dataset

#Splitting the training and test set
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.20, random_state=1)
#Splitting the train set into  dev set and training set
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.20, random_state=1)

# Checking the shape
print(X_train.shape)
print(X_val.shape)
print(y_train.shape)
print(y_val.shape)

(6400, 12)
(1600, 12)
(6400,)
(1600,)

scaler = preprocessing.MinMaxScaler()
# MinMaxScalar has been used here. You can go ahead and use the other scalars available and chcek the effect on the results.
#fitting the transform on test and train separately
X_train = scaler.fit_transform(X_train)
X_test = scaler.fit_transform(X_test)
X_val = scaler.fit_transform(X_val)

# create target scaler object

target_scaler = preprocessing.MinMaxScaler()
y_train = target_scaler.fit_transform(y_train.values.reshape(-1,1))
y_test = target_scaler.fit_transform(y_test.values.reshape(-1,1))
y_val=target_scaler.fit_transform(y_val.values.reshape(-1,1))

#Printing the data after normalization
X_val

array([[0.22304535, 1.        , 0.12162164, ..., 0.        , 0.        ,
        1.        ],
       [0.05264282, 0.564     , 0.43243244, ..., 1.        , 0.        ,
        0.        ],
       [0.5979347 , 0.17799997, 0.24324325, ..., 0.        , 0.        ,
        1.        ],
       ...,
       [0.5916748 , 0.15600002, 0.5945946 , ..., 0.        , 0.        ,
        0.        ],
       [0.4633026 , 0.658     , 0.13513514, ..., 0.        , 1.        ,
        0.        ],
       [0.5828018 , 0.65400004, 0.27027026, ..., 0.        , 0.        ,
        1.        ]], dtype=float32)

# let us now convert the data elements into tensors as we need tensors to be fed into different tensorflow based operations
#X-train and X_test were converted to numpy arrays while transformations while the other two need to be transformed into numpy arrays.
X_train_scale=tf.convert_to_tensor(X_train)
y_train_scale=tf.convert_to_tensor(y_train)
X_test_scale=tf.convert_to_tensor(X_test)
y_test_scale=tf.convert_to_tensor(y_test)
X_val_scale=tf.convert_to_tensor(X_val)
y_val_scale=tf.convert_to_tensor(y_val)

#printing the shape of training tensor
X_train_scale.shape[1]

12

Model building

• Build Neural Network

• Predicting customer is not going to churn and satisfied while customer is looking to churn

• Predicting customer is going to churn and while customer is not looking to churn

• Both are important for us to minimize, the bank would want the F1 Score evaluation metric to be maximized/ Hence, the focus should be on increasing the F1 score rather than focusing on just one metric i.e. Recall or Precision.

import numpy as np
import pandas as pd
from tensorflow.keras.utils import to_categorical
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
from matplotlib import pyplot
from scipy import stats
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import explained_variance_score, mean_squared_error, r2_score, mean_absolute_error
from collections import Counter
from keras import callbacks
import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout,InputLayer
from tensorflow.keras.models import Sequential
from keras.layers import BatchNormalization
from tensorflow.keras import regularizers
from tensorflow.keras import backend
from random import shuffle
from keras.callbacks import ModelCheckpoint
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.optimizers import RMSprop
import warnings
warnings.filterwarnings("ignore")

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Model 1

# Initializing the ANN
model = Sequential()
# The amount of nodes (dimensions) in hidden layer should be the average of input and output layers, in this case 64.
# This adds the input layer (by specifying input dimension) AND the first hidden layer (units)
model.add(Dense(64,activation = 'relu',input_shape=(X_train_scale.shape[1],)))
#Add 1st hidden layer
model.add(Dense(32, activation='relu'))
# Adding the output layer
# Notice that we do not need to specify input dim. 
# we have an output of 1 node, which is the the desired dimensions of our output (stay with the bank or not)
# We use the sigmoid because we want probability outcomes
model.add(Dense(1, activation = 'sigmoid')) 

# Create optimizer with default learning rate
# Compile the model
model.compile(optimizer='SGD', loss='binary_crossentropy', metrics=['accuracy'])

history=model.fit(X_train_scale, y_train_scale,           
          validation_split=0.2,
          epochs=100,
          batch_size=32,verbose=1)

Epoch 1/100
160/160 [==============================] - 1s 2ms/step - loss: 0.5375 - accuracy: 0.7904 - val_loss: 0.5083 - val_accuracy: 0.7906
Epoch 2/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4924 - accuracy: 0.8000 - val_loss: 0.5023 - val_accuracy: 0.7906
Epoch 3/100
160/160 [==============================] - 0s 991us/step - loss: 0.4869 - accuracy: 0.8000 - val_loss: 0.4978 - val_accuracy: 0.7906
Epoch 4/100
160/160 [==============================] - 0s 978us/step - loss: 0.4825 - accuracy: 0.8000 - val_loss: 0.4930 - val_accuracy: 0.7906
Epoch 5/100
160/160 [==============================] - 0s 885us/step - loss: 0.4786 - accuracy: 0.8000 - val_loss: 0.4896 - val_accuracy: 0.7906
Epoch 6/100
160/160 [==============================] - 0s 918us/step - loss: 0.4754 - accuracy: 0.8000 - val_loss: 0.4860 - val_accuracy: 0.7906
Epoch 7/100
160/160 [==============================] - 0s 971us/step - loss: 0.4724 - accuracy: 0.8000 - val_loss: 0.4833 - val_accuracy: 0.7906
Epoch 8/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4700 - accuracy: 0.8000 - val_loss: 0.4809 - val_accuracy: 0.7906
Epoch 9/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4676 - accuracy: 0.8000 - val_loss: 0.4781 - val_accuracy: 0.7906
Epoch 10/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4657 - accuracy: 0.8000 - val_loss: 0.4759 - val_accuracy: 0.7906
Epoch 11/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4639 - accuracy: 0.8000 - val_loss: 0.4741 - val_accuracy: 0.7906
Epoch 12/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4622 - accuracy: 0.8000 - val_loss: 0.4726 - val_accuracy: 0.7906
Epoch 13/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4608 - accuracy: 0.8000 - val_loss: 0.4708 - val_accuracy: 0.7906
Epoch 14/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4591 - accuracy: 0.8000 - val_loss: 0.4691 - val_accuracy: 0.7906
Epoch 15/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4577 - accuracy: 0.8000 - val_loss: 0.4679 - val_accuracy: 0.7906
Epoch 16/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4564 - accuracy: 0.8000 - val_loss: 0.4662 - val_accuracy: 0.7906
Epoch 17/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4550 - accuracy: 0.7998 - val_loss: 0.4645 - val_accuracy: 0.7906
Epoch 18/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4534 - accuracy: 0.8002 - val_loss: 0.4643 - val_accuracy: 0.7906
Epoch 19/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4522 - accuracy: 0.8006 - val_loss: 0.4617 - val_accuracy: 0.7906
Epoch 20/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4509 - accuracy: 0.8008 - val_loss: 0.4605 - val_accuracy: 0.7930
Epoch 21/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4495 - accuracy: 0.8020 - val_loss: 0.4586 - val_accuracy: 0.7937
Epoch 22/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4481 - accuracy: 0.8041 - val_loss: 0.4583 - val_accuracy: 0.7930
Epoch 23/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4469 - accuracy: 0.8039 - val_loss: 0.4573 - val_accuracy: 0.7937
Epoch 24/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4456 - accuracy: 0.8041 - val_loss: 0.4552 - val_accuracy: 0.7961
Epoch 25/100
160/160 [==============================] - 0s 981us/step - loss: 0.4443 - accuracy: 0.8043 - val_loss: 0.4528 - val_accuracy: 0.7992
Epoch 26/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4431 - accuracy: 0.8053 - val_loss: 0.4517 - val_accuracy: 0.7977
Epoch 27/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4416 - accuracy: 0.8045 - val_loss: 0.4500 - val_accuracy: 0.8023
Epoch 28/100
160/160 [==============================] - 0s 949us/step - loss: 0.4401 - accuracy: 0.8072 - val_loss: 0.4501 - val_accuracy: 0.7977
Epoch 29/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4394 - accuracy: 0.8082 - val_loss: 0.4479 - val_accuracy: 0.7992
Epoch 30/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4377 - accuracy: 0.8082 - val_loss: 0.4472 - val_accuracy: 0.8000
Epoch 31/100
160/160 [==============================] - 0s 956us/step - loss: 0.4363 - accuracy: 0.8102 - val_loss: 0.4465 - val_accuracy: 0.8016
Epoch 32/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4356 - accuracy: 0.8086 - val_loss: 0.4437 - val_accuracy: 0.8094
Epoch 33/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4345 - accuracy: 0.8111 - val_loss: 0.4435 - val_accuracy: 0.8062
Epoch 34/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4335 - accuracy: 0.8121 - val_loss: 0.4424 - val_accuracy: 0.8094
Epoch 35/100
160/160 [==============================] - 0s 955us/step - loss: 0.4324 - accuracy: 0.8107 - val_loss: 0.4408 - val_accuracy: 0.8141
Epoch 36/100
160/160 [==============================] - 0s 980us/step - loss: 0.4312 - accuracy: 0.8104 - val_loss: 0.4398 - val_accuracy: 0.8156
Epoch 37/100
160/160 [==============================] - 0s 972us/step - loss: 0.4304 - accuracy: 0.8129 - val_loss: 0.4395 - val_accuracy: 0.8148
Epoch 38/100
160/160 [==============================] - 0s 879us/step - loss: 0.4294 - accuracy: 0.8119 - val_loss: 0.4392 - val_accuracy: 0.8141
Epoch 39/100
160/160 [==============================] - 0s 864us/step - loss: 0.4289 - accuracy: 0.8139 - val_loss: 0.4383 - val_accuracy: 0.8148
Epoch 40/100
160/160 [==============================] - 0s 814us/step - loss: 0.4279 - accuracy: 0.8145 - val_loss: 0.4391 - val_accuracy: 0.8109
Epoch 41/100
160/160 [==============================] - 0s 862us/step - loss: 0.4273 - accuracy: 0.8135 - val_loss: 0.4381 - val_accuracy: 0.8102
Epoch 42/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4267 - accuracy: 0.8141 - val_loss: 0.4378 - val_accuracy: 0.8109
Epoch 43/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4261 - accuracy: 0.8156 - val_loss: 0.4359 - val_accuracy: 0.8156
Epoch 44/100
160/160 [==============================] - 0s 975us/step - loss: 0.4257 - accuracy: 0.8158 - val_loss: 0.4352 - val_accuracy: 0.8164
Epoch 45/100
160/160 [==============================] - 0s 864us/step - loss: 0.4248 - accuracy: 0.8154 - val_loss: 0.4362 - val_accuracy: 0.8156
Epoch 46/100
160/160 [==============================] - 0s 971us/step - loss: 0.4243 - accuracy: 0.8158 - val_loss: 0.4342 - val_accuracy: 0.8148
Epoch 47/100
160/160 [==============================] - 0s 898us/step - loss: 0.4239 - accuracy: 0.8172 - val_loss: 0.4338 - val_accuracy: 0.8172
Epoch 48/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4232 - accuracy: 0.8180 - val_loss: 0.4349 - val_accuracy: 0.8164
Epoch 49/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4224 - accuracy: 0.8164 - val_loss: 0.4337 - val_accuracy: 0.8141
Epoch 50/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4226 - accuracy: 0.8164 - val_loss: 0.4329 - val_accuracy: 0.8180
Epoch 51/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4221 - accuracy: 0.8174 - val_loss: 0.4338 - val_accuracy: 0.8172
Epoch 52/100
160/160 [==============================] - 0s 989us/step - loss: 0.4216 - accuracy: 0.8180 - val_loss: 0.4331 - val_accuracy: 0.8172
Epoch 53/100
160/160 [==============================] - 0s 873us/step - loss: 0.4212 - accuracy: 0.8164 - val_loss: 0.4324 - val_accuracy: 0.8180
Epoch 54/100
160/160 [==============================] - 0s 958us/step - loss: 0.4205 - accuracy: 0.8162 - val_loss: 0.4328 - val_accuracy: 0.8180
Epoch 55/100
160/160 [==============================] - 0s 942us/step - loss: 0.4202 - accuracy: 0.8170 - val_loss: 0.4317 - val_accuracy: 0.8188
Epoch 56/100
160/160 [==============================] - 0s 940us/step - loss: 0.4198 - accuracy: 0.8180 - val_loss: 0.4325 - val_accuracy: 0.8172
Epoch 57/100
160/160 [==============================] - 0s 985us/step - loss: 0.4196 - accuracy: 0.8172 - val_loss: 0.4316 - val_accuracy: 0.8219
Epoch 58/100
160/160 [==============================] - 0s 946us/step - loss: 0.4187 - accuracy: 0.8180 - val_loss: 0.4310 - val_accuracy: 0.8164
Epoch 59/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4186 - accuracy: 0.8180 - val_loss: 0.4308 - val_accuracy: 0.8172
Epoch 60/100
160/160 [==============================] - 0s 847us/step - loss: 0.4182 - accuracy: 0.8164 - val_loss: 0.4320 - val_accuracy: 0.8188
Epoch 61/100
160/160 [==============================] - 0s 839us/step - loss: 0.4177 - accuracy: 0.8207 - val_loss: 0.4320 - val_accuracy: 0.8164
Epoch 62/100
160/160 [==============================] - 0s 903us/step - loss: 0.4174 - accuracy: 0.8195 - val_loss: 0.4300 - val_accuracy: 0.8188
Epoch 63/100
160/160 [==============================] - 0s 899us/step - loss: 0.4161 - accuracy: 0.8193 - val_loss: 0.4302 - val_accuracy: 0.8211
Epoch 64/100
160/160 [==============================] - 0s 962us/step - loss: 0.4162 - accuracy: 0.8207 - val_loss: 0.4293 - val_accuracy: 0.8203
Epoch 65/100
160/160 [==============================] - 0s 836us/step - loss: 0.4154 - accuracy: 0.8223 - val_loss: 0.4291 - val_accuracy: 0.8227
Epoch 66/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4151 - accuracy: 0.8209 - val_loss: 0.4289 - val_accuracy: 0.8203
Epoch 67/100
160/160 [==============================] - 0s 860us/step - loss: 0.4148 - accuracy: 0.8217 - val_loss: 0.4297 - val_accuracy: 0.8242
Epoch 68/100
160/160 [==============================] - 0s 873us/step - loss: 0.4143 - accuracy: 0.8223 - val_loss: 0.4295 - val_accuracy: 0.8242
Epoch 69/100
160/160 [==============================] - 0s 924us/step - loss: 0.4132 - accuracy: 0.8242 - val_loss: 0.4312 - val_accuracy: 0.8203
Epoch 70/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4132 - accuracy: 0.8244 - val_loss: 0.4276 - val_accuracy: 0.8227
Epoch 71/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4125 - accuracy: 0.8258 - val_loss: 0.4269 - val_accuracy: 0.8250
Epoch 72/100
160/160 [==============================] - 0s 875us/step - loss: 0.4117 - accuracy: 0.8266 - val_loss: 0.4265 - val_accuracy: 0.8250
Epoch 73/100
160/160 [==============================] - 0s 906us/step - loss: 0.4109 - accuracy: 0.8254 - val_loss: 0.4261 - val_accuracy: 0.8273
Epoch 74/100
160/160 [==============================] - 0s 897us/step - loss: 0.4105 - accuracy: 0.8273 - val_loss: 0.4261 - val_accuracy: 0.8250
Epoch 75/100
160/160 [==============================] - 0s 837us/step - loss: 0.4096 - accuracy: 0.8293 - val_loss: 0.4278 - val_accuracy: 0.8242
Epoch 76/100
160/160 [==============================] - 0s 921us/step - loss: 0.4090 - accuracy: 0.8293 - val_loss: 0.4256 - val_accuracy: 0.8289
Epoch 77/100
160/160 [==============================] - 0s 1ms/step - loss: 0.4084 - accuracy: 0.8285 - val_loss: 0.4245 - val_accuracy: 0.8289
Epoch 78/100
160/160 [==============================] - 0s 941us/step - loss: 0.4077 - accuracy: 0.8283 - val_loss: 0.4236 - val_accuracy: 0.8281
Epoch 79/100
160/160 [==============================] - 0s 871us/step - loss: 0.4068 - accuracy: 0.8314 - val_loss: 0.4230 - val_accuracy: 0.8305
Epoch 80/100
160/160 [==============================] - 0s 973us/step - loss: 0.4061 - accuracy: 0.8309 - val_loss: 0.4242 - val_accuracy: 0.8266
Epoch 81/100
160/160 [==============================] - 0s 942us/step - loss: 0.4051 - accuracy: 0.8322 - val_loss: 0.4214 - val_accuracy: 0.8297
Epoch 82/100
160/160 [==============================] - 0s 877us/step - loss: 0.4043 - accuracy: 0.8322 - val_loss: 0.4220 - val_accuracy: 0.8297
Epoch 83/100
160/160 [==============================] - 0s 933us/step - loss: 0.4036 - accuracy: 0.8326 - val_loss: 0.4198 - val_accuracy: 0.8320
Epoch 84/100
160/160 [==============================] - 0s 861us/step - loss: 0.4024 - accuracy: 0.8320 - val_loss: 0.4210 - val_accuracy: 0.8289
Epoch 85/100
160/160 [==============================] - 0s 878us/step - loss: 0.4015 - accuracy: 0.8338 - val_loss: 0.4186 - val_accuracy: 0.8313
Epoch 86/100
160/160 [==============================] - 0s 887us/step - loss: 0.4004 - accuracy: 0.8338 - val_loss: 0.4171 - val_accuracy: 0.8328
Epoch 87/100
160/160 [==============================] - 0s 960us/step - loss: 0.3995 - accuracy: 0.8334 - val_loss: 0.4164 - val_accuracy: 0.8344
Epoch 88/100
160/160 [==============================] - 0s 964us/step - loss: 0.3988 - accuracy: 0.8326 - val_loss: 0.4167 - val_accuracy: 0.8320
Epoch 89/100
160/160 [==============================] - 0s 949us/step - loss: 0.3979 - accuracy: 0.8342 - val_loss: 0.4145 - val_accuracy: 0.8313
Epoch 90/100
160/160 [==============================] - 0s 974us/step - loss: 0.3967 - accuracy: 0.8375 - val_loss: 0.4136 - val_accuracy: 0.8336
Epoch 91/100
160/160 [==============================] - 0s 880us/step - loss: 0.3954 - accuracy: 0.8371 - val_loss: 0.4185 - val_accuracy: 0.8305
Epoch 92/100
160/160 [==============================] - 0s 887us/step - loss: 0.3944 - accuracy: 0.8367 - val_loss: 0.4115 - val_accuracy: 0.8352
Epoch 93/100
160/160 [==============================] - 0s 897us/step - loss: 0.3936 - accuracy: 0.8359 - val_loss: 0.4106 - val_accuracy: 0.8352
Epoch 94/100
160/160 [==============================] - 0s 945us/step - loss: 0.3927 - accuracy: 0.8393 - val_loss: 0.4094 - val_accuracy: 0.8352
Epoch 95/100
160/160 [==============================] - 0s 1ms/step - loss: 0.3914 - accuracy: 0.8369 - val_loss: 0.4085 - val_accuracy: 0.8359
Epoch 96/100
160/160 [==============================] - 0s 1ms/step - loss: 0.3902 - accuracy: 0.8402 - val_loss: 0.4074 - val_accuracy: 0.8375
Epoch 97/100
160/160 [==============================] - 0s 992us/step - loss: 0.3891 - accuracy: 0.8391 - val_loss: 0.4081 - val_accuracy: 0.8344
Epoch 98/100
160/160 [==============================] - 0s 867us/step - loss: 0.3874 - accuracy: 0.8406 - val_loss: 0.4056 - val_accuracy: 0.8367
Epoch 99/100
160/160 [==============================] - 0s 841us/step - loss: 0.3862 - accuracy: 0.8408 - val_loss: 0.4100 - val_accuracy: 0.8383
Epoch 100/100
160/160 [==============================] - 0s 875us/step - loss: 0.3857 - accuracy: 0.8428 - val_loss: 0.4031 - val_accuracy: 0.8398

# Capturing learning history per epoch
hist  = pd.DataFrame(history.history)
hist['epoch'] = history.epoch

# Plotting accuracy at different epochs
plt.plot(hist['loss'])
plt.plot(hist['val_loss'])
plt.legend(("train" , "valid") , loc =0)

#Printing results
results = model.evaluate(X_test_scale, y_test_scale)

63/63 [==============================] - 0s 1ms/step - loss: 0.3891 - accuracy: 0.8375

y_pred_scale=model.predict(X_test_scale)
y_pred_scale = (y_pred_scale > 0.5)
y_pred_scale

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [False]])

def make_confusion_matrix(cf,
                          group_names=None,
                          categories='auto',
                          count=True,
                          percent=True,
                          cbar=True,
                          xyticks=True,
                          xyplotlabels=True,
                          sum_stats=True,
                          figsize=None,
                          cmap='Blues',
                          title=None):
    '''
    This function will make a pretty plot of an sklearn Confusion Matrix cm using a Seaborn heatmap visualization.
    Arguments
    '''


    # CODE TO GENERATE TEXT INSIDE EACH SQUARE
    blanks = ['' for i in range(cf.size)]

    if group_names and len(group_names)==cf.size:
        group_labels = ["{}\n".format(value) for value in group_names]
    else:
        group_labels = blanks

    if count:
        group_counts = ["{0:0.0f}\n".format(value) for value in cf.flatten()]
    else:
        group_counts = blanks

    if percent:
        group_percentages = ["{0:.2%}".format(value) for value in cf.flatten()/np.sum(cf)]
    else:
        group_percentages = blanks

    box_labels = [f"{v1}{v2}{v3}".strip() for v1, v2, v3 in zip(group_labels,group_counts,group_percentages)]
    box_labels = np.asarray(box_labels).reshape(cf.shape[0],cf.shape[1])


    # CODE TO GENERATE SUMMARY STATISTICS & TEXT FOR SUMMARY STATS
    if sum_stats:
        #Accuracy is sum of diagonal divided by total observations
        accuracy  = np.trace(cf) / float(np.sum(cf))



    # SET FIGURE PARAMETERS ACCORDING TO OTHER ARGUMENTS
    if figsize==None:
        #Get default figure size if not set
        figsize = plt.rcParams.get('figure.figsize')

    if xyticks==False:
        #Do not show categories if xyticks is False
        categories=False


    # MAKE THE HEATMAP VISUALIZATION
    plt.figure(figsize=figsize)
    sns.heatmap(cf,annot=box_labels,fmt="",cmap=cmap,cbar=cbar,xticklabels=categories,yticklabels=categories)

    
    if title:
        plt.title(title)

#Calculating the confusion matrix 
from sklearn.metrics import confusion_matrix
cm=confusion_matrix(y_test_scale, y_pred_scale)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm, 
                      group_names=labels,
                      categories=categories, 
                      cmap='Blues')

#Accuracy as per the classification report 
from sklearn import metrics
cr=metrics.classification_report(y_test_scale,y_pred_scale)
print(cr)

              precision    recall  f1-score   support

         0.0       0.85      0.96      0.90      1585
         1.0       0.72      0.35      0.47       415

    accuracy                           0.84      2000
   macro avg       0.79      0.66      0.69      2000
weighted avg       0.82      0.84      0.81      2000

1) Imbalanced dataset: As you have seen in the EDA, this dataset is imbalanced, and it contains more examples that belong to the 0 class.

2) Decision Threshold: Due to the imbalanced dataset, we can use ROC-AUC to find the optimal threshold and use the same for prediction.

Let's try to change the optimizer, tune the decision threshold, increase the layers and configure some other hyperparameters accordingly, in order to improve the model's performance.

Model Performance Improvement

-Comment on which metric is right for model performance evaluation and why? - Find the optimal threshold using ROC-AUC curves - Comment on model performance - Can model performance be improved? check and comment - Build another model to implement these improvements - Include all the model which were trained to reach at the final one

Model 2

backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

model1 = Sequential()
      #Adding the hidden and output layers
model1.add(Dense(256,activation='relu',kernel_initializer='he_uniform',input_dim = X_train_scale.shape[1]))
model1.add(Dense(128,activation='relu',kernel_initializer='he_uniform'))
model1.add(Dense(64,activation='relu',kernel_initializer='he_uniform'))
model1.add(Dense(32,activation='relu',kernel_initializer='he_uniform'))
model1.add(Dense(1, activation = 'sigmoid'))
      #Compiling the ANN with Adam optimizer and binary cross entropy loss function 
optimizer = tf.keras.optimizers.Adam(0.001)
model1.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])

model1.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 256)               3328      
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 dense_3 (Dense)             (None, 32)                2080      
                                                                 
 dense_4 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 46,593
Trainable params: 46,593
Non-trainable params: 0
_________________________________________________________________

history1 = model1.fit(X_train_scale,y_train_scale,batch_size=64,epochs=50,verbose=1,validation_split = 0.2)

Epoch 1/50
80/80 [==============================] - 1s 2ms/step - loss: 0.4775 - accuracy: 0.7953 - val_loss: 0.4749 - val_accuracy: 0.7922
Epoch 2/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4424 - accuracy: 0.8135 - val_loss: 0.4359 - val_accuracy: 0.8297
Epoch 3/50
80/80 [==============================] - 0s 1ms/step - loss: 0.4069 - accuracy: 0.8295 - val_loss: 0.4292 - val_accuracy: 0.8203
Epoch 4/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3747 - accuracy: 0.8469 - val_loss: 0.3940 - val_accuracy: 0.8422
Epoch 5/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3535 - accuracy: 0.8547 - val_loss: 0.3732 - val_accuracy: 0.8391
Epoch 6/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3541 - accuracy: 0.8521 - val_loss: 0.3737 - val_accuracy: 0.8414
Epoch 7/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3364 - accuracy: 0.8586 - val_loss: 0.3691 - val_accuracy: 0.8492
Epoch 8/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3321 - accuracy: 0.8625 - val_loss: 0.3805 - val_accuracy: 0.8398
Epoch 9/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3292 - accuracy: 0.8615 - val_loss: 0.3649 - val_accuracy: 0.8453
Epoch 10/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3225 - accuracy: 0.8658 - val_loss: 0.4037 - val_accuracy: 0.8391
Epoch 11/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3218 - accuracy: 0.8666 - val_loss: 0.3649 - val_accuracy: 0.8469
Epoch 12/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3159 - accuracy: 0.8654 - val_loss: 0.3731 - val_accuracy: 0.8531
Epoch 13/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3108 - accuracy: 0.8721 - val_loss: 0.3700 - val_accuracy: 0.8367
Epoch 14/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3059 - accuracy: 0.8723 - val_loss: 0.3728 - val_accuracy: 0.8531
Epoch 15/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2964 - accuracy: 0.8754 - val_loss: 0.3746 - val_accuracy: 0.8492
Epoch 16/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2939 - accuracy: 0.8773 - val_loss: 0.3755 - val_accuracy: 0.8484
Epoch 17/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2948 - accuracy: 0.8764 - val_loss: 0.3780 - val_accuracy: 0.8438
Epoch 18/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2827 - accuracy: 0.8826 - val_loss: 0.4061 - val_accuracy: 0.8477
Epoch 19/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2815 - accuracy: 0.8811 - val_loss: 0.3864 - val_accuracy: 0.8508
Epoch 20/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2706 - accuracy: 0.8902 - val_loss: 0.3992 - val_accuracy: 0.8359
Epoch 21/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2719 - accuracy: 0.8891 - val_loss: 0.3996 - val_accuracy: 0.8523
Epoch 22/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2671 - accuracy: 0.8896 - val_loss: 0.3965 - val_accuracy: 0.8477
Epoch 23/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2601 - accuracy: 0.8906 - val_loss: 0.4130 - val_accuracy: 0.8398
Epoch 24/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2565 - accuracy: 0.8918 - val_loss: 0.4213 - val_accuracy: 0.8469
Epoch 25/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2533 - accuracy: 0.8949 - val_loss: 0.4119 - val_accuracy: 0.8297
Epoch 26/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2442 - accuracy: 0.8990 - val_loss: 0.4165 - val_accuracy: 0.8297
Epoch 27/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2451 - accuracy: 0.8984 - val_loss: 0.4197 - val_accuracy: 0.8414
Epoch 28/50
80/80 [==============================] - 0s 1ms/step - loss: 0.2370 - accuracy: 0.9033 - val_loss: 0.4409 - val_accuracy: 0.8414
Epoch 29/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2358 - accuracy: 0.9029 - val_loss: 0.4318 - val_accuracy: 0.8430
Epoch 30/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2284 - accuracy: 0.9070 - val_loss: 0.4573 - val_accuracy: 0.8391
Epoch 31/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2217 - accuracy: 0.9082 - val_loss: 0.4575 - val_accuracy: 0.8398
Epoch 32/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2113 - accuracy: 0.9152 - val_loss: 0.4797 - val_accuracy: 0.8414
Epoch 33/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2122 - accuracy: 0.9182 - val_loss: 0.4857 - val_accuracy: 0.8406
Epoch 34/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2070 - accuracy: 0.9176 - val_loss: 0.4931 - val_accuracy: 0.8430
Epoch 35/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2026 - accuracy: 0.9178 - val_loss: 0.5099 - val_accuracy: 0.8391
Epoch 36/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1972 - accuracy: 0.9205 - val_loss: 0.5053 - val_accuracy: 0.8406
Epoch 37/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1976 - accuracy: 0.9197 - val_loss: 0.5166 - val_accuracy: 0.8156
Epoch 38/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1938 - accuracy: 0.9203 - val_loss: 0.5246 - val_accuracy: 0.8344
Epoch 39/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1834 - accuracy: 0.9225 - val_loss: 0.5415 - val_accuracy: 0.8328
Epoch 40/50
80/80 [==============================] - 0s 1ms/step - loss: 0.1765 - accuracy: 0.9305 - val_loss: 0.5524 - val_accuracy: 0.8281
Epoch 41/50
80/80 [==============================] - 0s 1ms/step - loss: 0.1669 - accuracy: 0.9330 - val_loss: 0.5779 - val_accuracy: 0.8406
Epoch 42/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1740 - accuracy: 0.9301 - val_loss: 0.5669 - val_accuracy: 0.8438
Epoch 43/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1609 - accuracy: 0.9342 - val_loss: 0.5679 - val_accuracy: 0.8234
Epoch 44/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1589 - accuracy: 0.9344 - val_loss: 0.6220 - val_accuracy: 0.8250
Epoch 45/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1566 - accuracy: 0.9359 - val_loss: 0.6145 - val_accuracy: 0.8250
Epoch 46/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1492 - accuracy: 0.9406 - val_loss: 0.6397 - val_accuracy: 0.8188
Epoch 47/50
80/80 [==============================] - 0s 1ms/step - loss: 0.1406 - accuracy: 0.9449 - val_loss: 0.6310 - val_accuracy: 0.8227
Epoch 48/50
80/80 [==============================] - 0s 1ms/step - loss: 0.1379 - accuracy: 0.9447 - val_loss: 0.6821 - val_accuracy: 0.8148
Epoch 49/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1428 - accuracy: 0.9404 - val_loss: 0.6652 - val_accuracy: 0.8195
Epoch 50/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1391 - accuracy: 0.9434 - val_loss: 0.6665 - val_accuracy: 0.8125

#Plotting Train Loss vs Validation Loss
plt.plot(history1.history['loss'])
plt.plot(history1.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

from sklearn.metrics import roc_curve

from matplotlib import pyplot


# predict probabilities
yhat1 = model1.predict(X_test_scale)
# keep probabilities for the positive outcome only
yhat1 = yhat1[:, 0]
# calculate roc curves
fpr, tpr, thresholds1 = roc_curve(y_test_scale, yhat1)
# calculate the g-mean for each threshold
gmeans1 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans1)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds1[ix], gmeans1[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='NO Churn')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()

Best Threshold=0.085891, G-Mean=0.730

• G-Mean = sqrt(Sensitivity * Specificity)
• select the threshold with the largest G-Mean value.

#Predicting the results using best as a threshold
y_pred_e1=model1.predict(X_test_scale)
y_pred_e1 = (y_pred_e1 > thresholds1[ix])
y_pred_e1

array([[False],
       [False],
       [ True],
       ...,
       [False],
       [False],
       [ True]])

#Calculating the confusion matrix 
from sklearn.metrics import confusion_matrix
cm1=confusion_matrix(y_test_scale, y_pred_e1)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm1, 
                      group_names=labels,
                      categories=categories, 
                      cmap='Blues')

#Accuracy as per the classification report 
from sklearn import metrics
cr1=metrics.classification_report(y_test_scale,y_pred_e1)
print(cr1)

              precision    recall  f1-score   support

         0.0       0.91      0.75      0.82      1585
         1.0       0.42      0.71      0.53       415

    accuracy                           0.74      2000
   macro avg       0.67      0.73      0.68      2000
weighted avg       0.81      0.74      0.76      2000

Model 3

backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

model2 = Sequential()
model2.add(Dense(128,activation='relu',input_dim = X_train_scale.shape[1]))
model2.add(BatchNormalization())
model2.add(Dense(64,activation='relu',kernel_initializer='he_uniform'))
model2.add(BatchNormalization())
model2.add(Dense(32,activation='relu',kernel_initializer='he_uniform'))
model2.add(Dense(1, activation = 'sigmoid'))

model2.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 128)               1664      
                                                                 
 batch_normalization (BatchN  (None, 128)              512       
 ormalization)                                                   
                                                                 
 dense_1 (Dense)             (None, 64)                8256      
                                                                 
 batch_normalization_1 (Batc  (None, 64)               256       
 hNormalization)                                                 
                                                                 
 dense_2 (Dense)             (None, 32)                2080      
                                                                 
 dense_3 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 12,801
Trainable params: 12,417
Non-trainable params: 384
_________________________________________________________________

optimizer = tf.keras.optimizers.Adam(0.001)
model2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])

history_2 = model2.fit(X_train_scale,y_train_scale,batch_size=64,epochs=50,verbose=1,validation_split = 0.2)

Epoch 1/50
80/80 [==============================] - 1s 3ms/step - loss: 0.5102 - accuracy: 0.7555 - val_loss: 0.5450 - val_accuracy: 0.7906
Epoch 2/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4003 - accuracy: 0.8330 - val_loss: 0.4895 - val_accuracy: 0.7906
Epoch 3/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3662 - accuracy: 0.8480 - val_loss: 0.4553 - val_accuracy: 0.7937
Epoch 4/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3447 - accuracy: 0.8562 - val_loss: 0.4186 - val_accuracy: 0.8117
Epoch 5/50
80/80 [==============================] - 0s 1ms/step - loss: 0.3331 - accuracy: 0.8648 - val_loss: 0.4145 - val_accuracy: 0.8180
Epoch 6/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3308 - accuracy: 0.8609 - val_loss: 0.3883 - val_accuracy: 0.8422
Epoch 7/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3163 - accuracy: 0.8686 - val_loss: 0.3900 - val_accuracy: 0.8445
Epoch 8/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3125 - accuracy: 0.8652 - val_loss: 0.3821 - val_accuracy: 0.8555
Epoch 9/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3064 - accuracy: 0.8703 - val_loss: 0.3782 - val_accuracy: 0.8594
Epoch 10/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3058 - accuracy: 0.8686 - val_loss: 0.3788 - val_accuracy: 0.8367
Epoch 11/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3006 - accuracy: 0.8740 - val_loss: 0.3817 - val_accuracy: 0.8484
Epoch 12/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2893 - accuracy: 0.8801 - val_loss: 0.3886 - val_accuracy: 0.8508
Epoch 13/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2952 - accuracy: 0.8713 - val_loss: 0.3875 - val_accuracy: 0.8477
Epoch 14/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2880 - accuracy: 0.8783 - val_loss: 0.3935 - val_accuracy: 0.8398
Epoch 15/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2828 - accuracy: 0.8797 - val_loss: 0.4020 - val_accuracy: 0.8516
Epoch 16/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2818 - accuracy: 0.8809 - val_loss: 0.4016 - val_accuracy: 0.8414
Epoch 17/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2754 - accuracy: 0.8885 - val_loss: 0.4026 - val_accuracy: 0.8469
Epoch 18/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2769 - accuracy: 0.8809 - val_loss: 0.4024 - val_accuracy: 0.8492
Epoch 19/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2675 - accuracy: 0.8893 - val_loss: 0.3993 - val_accuracy: 0.8492
Epoch 20/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2624 - accuracy: 0.8902 - val_loss: 0.4128 - val_accuracy: 0.8477
Epoch 21/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2643 - accuracy: 0.8887 - val_loss: 0.4213 - val_accuracy: 0.8438
Epoch 22/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2598 - accuracy: 0.8943 - val_loss: 0.4255 - val_accuracy: 0.8422
Epoch 23/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2583 - accuracy: 0.8900 - val_loss: 0.4244 - val_accuracy: 0.8430
Epoch 24/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2526 - accuracy: 0.8926 - val_loss: 0.4193 - val_accuracy: 0.8391
Epoch 25/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2447 - accuracy: 0.8975 - val_loss: 0.4259 - val_accuracy: 0.8352
Epoch 26/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2411 - accuracy: 0.8998 - val_loss: 0.4380 - val_accuracy: 0.8375
Epoch 27/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2416 - accuracy: 0.8969 - val_loss: 0.4261 - val_accuracy: 0.8344
Epoch 28/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2386 - accuracy: 0.8975 - val_loss: 0.4288 - val_accuracy: 0.8422
Epoch 29/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2331 - accuracy: 0.9033 - val_loss: 0.4393 - val_accuracy: 0.8406
Epoch 30/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2299 - accuracy: 0.9051 - val_loss: 0.4607 - val_accuracy: 0.8438
Epoch 31/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2283 - accuracy: 0.9014 - val_loss: 0.4442 - val_accuracy: 0.8359
Epoch 32/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2228 - accuracy: 0.9021 - val_loss: 0.4685 - val_accuracy: 0.8320
Epoch 33/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2219 - accuracy: 0.9062 - val_loss: 0.4522 - val_accuracy: 0.8289
Epoch 34/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2178 - accuracy: 0.9100 - val_loss: 0.4701 - val_accuracy: 0.8406
Epoch 35/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2144 - accuracy: 0.9111 - val_loss: 0.4706 - val_accuracy: 0.8313
Epoch 36/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2074 - accuracy: 0.9129 - val_loss: 0.4833 - val_accuracy: 0.8313
Epoch 37/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2163 - accuracy: 0.9080 - val_loss: 0.4659 - val_accuracy: 0.8297
Epoch 38/50
80/80 [==============================] - 0s 1ms/step - loss: 0.2075 - accuracy: 0.9094 - val_loss: 0.4796 - val_accuracy: 0.8250
Epoch 39/50
80/80 [==============================] - 0s 1ms/step - loss: 0.2038 - accuracy: 0.9137 - val_loss: 0.4804 - val_accuracy: 0.8391
Epoch 40/50
80/80 [==============================] - 0s 1ms/step - loss: 0.2027 - accuracy: 0.9121 - val_loss: 0.4846 - val_accuracy: 0.8328
Epoch 41/50
80/80 [==============================] - 0s 1ms/step - loss: 0.1972 - accuracy: 0.9197 - val_loss: 0.4964 - val_accuracy: 0.8359
Epoch 42/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1941 - accuracy: 0.9182 - val_loss: 0.5089 - val_accuracy: 0.8219
Epoch 43/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1918 - accuracy: 0.9213 - val_loss: 0.5067 - val_accuracy: 0.8344
Epoch 44/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1879 - accuracy: 0.9242 - val_loss: 0.5357 - val_accuracy: 0.8289
Epoch 45/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1885 - accuracy: 0.9225 - val_loss: 0.5205 - val_accuracy: 0.8336
Epoch 46/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1853 - accuracy: 0.9234 - val_loss: 0.5238 - val_accuracy: 0.8250
Epoch 47/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1761 - accuracy: 0.9242 - val_loss: 0.5027 - val_accuracy: 0.8211
Epoch 48/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1753 - accuracy: 0.9268 - val_loss: 0.5400 - val_accuracy: 0.8273
Epoch 49/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1781 - accuracy: 0.9242 - val_loss: 0.5225 - val_accuracy: 0.8289
Epoch 50/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1725 - accuracy: 0.9240 - val_loss: 0.5405 - val_accuracy: 0.8273

#Plotting Train Loss vs Validation Loss
plt.plot(history_2.history['loss'])
plt.plot(history_2.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

• there is lot of noise and difference between train and test

from sklearn.metrics import roc_curve

from matplotlib import pyplot


# predict probabilities
yhat2 = model2.predict(X_test_scale)
# keep probabilities for the positive outcome only
yhat2 = yhat2[:, 0]
# calculate roc curves
fpr, tpr, thresholds2 = roc_curve(y_test_scale, yhat2)
# calculate the g-mean for each threshold
gmeans2 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans2)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds2[ix], gmeans2[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Skill')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()

Best Threshold=0.174853, G-Mean=0.741

y_pred_e2=model2.predict(X_test_scale)
y_pred_e2 = (y_pred_e2 > thresholds2[ix])
y_pred_e2

array([[False],
       [ True],
       [False],
       ...,
       [False],
       [False],
       [ True]])

#Calculating the confusion matrix 
from sklearn.metrics import confusion_matrix
cm2=confusion_matrix(y_test_scale, y_pred_e2)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm2, 
                      group_names=labels,
                      categories=categories, 
                      cmap='Blues')

#Accuracy as per the classification report 
from sklearn import metrics
cr3=metrics.classification_report(y_test_scale,y_pred_e2)
print(cr3)

              precision    recall  f1-score   support

         0.0       0.91      0.78      0.84      1585
         1.0       0.46      0.70      0.55       415

    accuracy                           0.77      2000
   macro avg       0.68      0.74      0.70      2000
weighted avg       0.82      0.77      0.78      2000

• The Train and Validation curves seem to show overfitting despite having a good F1 score and a low False Negative rate.

• Let's try to use the Dropout technique and check to see if it can reduce the False Negative rate.

Model 4

backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

model3 = Sequential()
model3.add(Dense(256,activation='relu',input_dim = X_train_scale.shape[1]))
model3.add(Dropout(0.2))
model3.add(Dense(128,activation='relu'))
model3.add(Dropout(0.2))
model3.add(Dense(64,activation='relu'))
model3.add(Dropout(0.2))
model3.add(Dense(32,activation='relu'))
model3.add(Dense(1, activation = 'sigmoid'))

model3.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 256)               3328      
                                                                 
 dropout (Dropout)           (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dropout_1 (Dropout)         (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 dropout_2 (Dropout)         (None, 64)                0         
                                                                 
 dense_3 (Dense)             (None, 32)                2080      
                                                                 
 dense_4 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 46,593
Trainable params: 46,593
Non-trainable params: 0
_________________________________________________________________

optimizer = tf.keras.optimizers.Adam(0.001)
model3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])

history_3 = model3.fit(X_train_scale,y_train_scale,batch_size=64,epochs=50,verbose=1,validation_split = 0.2)

Epoch 1/50
80/80 [==============================] - 1s 3ms/step - loss: 0.4967 - accuracy: 0.7922 - val_loss: 0.4701 - val_accuracy: 0.7906
Epoch 2/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4594 - accuracy: 0.8002 - val_loss: 0.4551 - val_accuracy: 0.7906
Epoch 3/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4447 - accuracy: 0.8043 - val_loss: 0.4337 - val_accuracy: 0.8188
Epoch 4/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4203 - accuracy: 0.8270 - val_loss: 0.4161 - val_accuracy: 0.8328
Epoch 5/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3962 - accuracy: 0.8328 - val_loss: 0.3929 - val_accuracy: 0.8406
Epoch 6/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3847 - accuracy: 0.8412 - val_loss: 0.3844 - val_accuracy: 0.8430
Epoch 7/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3713 - accuracy: 0.8486 - val_loss: 0.3743 - val_accuracy: 0.8469
Epoch 8/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3621 - accuracy: 0.8510 - val_loss: 0.3699 - val_accuracy: 0.8484
Epoch 9/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3548 - accuracy: 0.8535 - val_loss: 0.3661 - val_accuracy: 0.8430
Epoch 10/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3544 - accuracy: 0.8529 - val_loss: 0.3679 - val_accuracy: 0.8422
Epoch 11/50
80/80 [==============================] - 0s 1ms/step - loss: 0.3474 - accuracy: 0.8592 - val_loss: 0.3599 - val_accuracy: 0.8547
Epoch 12/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3496 - accuracy: 0.8562 - val_loss: 0.3650 - val_accuracy: 0.8539
Epoch 13/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3407 - accuracy: 0.8637 - val_loss: 0.3629 - val_accuracy: 0.8531
Epoch 14/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3353 - accuracy: 0.8639 - val_loss: 0.3665 - val_accuracy: 0.8539
Epoch 15/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3339 - accuracy: 0.8629 - val_loss: 0.3561 - val_accuracy: 0.8570
Epoch 16/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3283 - accuracy: 0.8670 - val_loss: 0.3550 - val_accuracy: 0.8586
Epoch 17/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3323 - accuracy: 0.8631 - val_loss: 0.3603 - val_accuracy: 0.8516
Epoch 18/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3259 - accuracy: 0.8676 - val_loss: 0.3795 - val_accuracy: 0.8570
Epoch 19/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3235 - accuracy: 0.8715 - val_loss: 0.3644 - val_accuracy: 0.8562
Epoch 20/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3169 - accuracy: 0.8719 - val_loss: 0.3667 - val_accuracy: 0.8516
Epoch 21/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3218 - accuracy: 0.8658 - val_loss: 0.3698 - val_accuracy: 0.8562
Epoch 22/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3207 - accuracy: 0.8691 - val_loss: 0.3613 - val_accuracy: 0.8586
Epoch 23/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3153 - accuracy: 0.8652 - val_loss: 0.3694 - val_accuracy: 0.8539
Epoch 24/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3205 - accuracy: 0.8682 - val_loss: 0.3811 - val_accuracy: 0.8523
Epoch 25/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3129 - accuracy: 0.8699 - val_loss: 0.3651 - val_accuracy: 0.8578
Epoch 26/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3090 - accuracy: 0.8729 - val_loss: 0.3630 - val_accuracy: 0.8531
Epoch 27/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3124 - accuracy: 0.8723 - val_loss: 0.3737 - val_accuracy: 0.8516
Epoch 28/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3051 - accuracy: 0.8742 - val_loss: 0.3738 - val_accuracy: 0.8508
Epoch 29/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3043 - accuracy: 0.8803 - val_loss: 0.3692 - val_accuracy: 0.8477
Epoch 30/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3085 - accuracy: 0.8734 - val_loss: 0.3682 - val_accuracy: 0.8508
Epoch 31/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3079 - accuracy: 0.8721 - val_loss: 0.3684 - val_accuracy: 0.8578
Epoch 32/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2988 - accuracy: 0.8783 - val_loss: 0.3685 - val_accuracy: 0.8523
Epoch 33/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3016 - accuracy: 0.8748 - val_loss: 0.3696 - val_accuracy: 0.8578
Epoch 34/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3026 - accuracy: 0.8750 - val_loss: 0.3793 - val_accuracy: 0.8570
Epoch 35/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2935 - accuracy: 0.8834 - val_loss: 0.3767 - val_accuracy: 0.8539
Epoch 36/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2942 - accuracy: 0.8775 - val_loss: 0.3826 - val_accuracy: 0.8500
Epoch 37/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2887 - accuracy: 0.8814 - val_loss: 0.3794 - val_accuracy: 0.8398
Epoch 38/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2870 - accuracy: 0.8836 - val_loss: 0.3915 - val_accuracy: 0.8547
Epoch 39/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2985 - accuracy: 0.8730 - val_loss: 0.3918 - val_accuracy: 0.8484
Epoch 40/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2905 - accuracy: 0.8824 - val_loss: 0.3794 - val_accuracy: 0.8453
Epoch 41/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2848 - accuracy: 0.8869 - val_loss: 0.3898 - val_accuracy: 0.8477
Epoch 42/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2863 - accuracy: 0.8834 - val_loss: 0.3919 - val_accuracy: 0.8438
Epoch 43/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2781 - accuracy: 0.8852 - val_loss: 0.3972 - val_accuracy: 0.8500
Epoch 44/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2800 - accuracy: 0.8863 - val_loss: 0.3922 - val_accuracy: 0.8414
Epoch 45/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2823 - accuracy: 0.8854 - val_loss: 0.3898 - val_accuracy: 0.8461
Epoch 46/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2778 - accuracy: 0.8846 - val_loss: 0.4007 - val_accuracy: 0.8438
Epoch 47/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2741 - accuracy: 0.8869 - val_loss: 0.3875 - val_accuracy: 0.8500
Epoch 48/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2731 - accuracy: 0.8883 - val_loss: 0.3946 - val_accuracy: 0.8539
Epoch 49/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2733 - accuracy: 0.8867 - val_loss: 0.3920 - val_accuracy: 0.8516
Epoch 50/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2783 - accuracy: 0.8857 - val_loss: 0.3987 - val_accuracy: 0.8469

#Plotting Train Loss vs Validation Loss
plt.plot(history_3.history['loss'])
plt.plot(history_3.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

from sklearn.metrics import roc_curve

from matplotlib import pyplot


# predict probabilities
yhat3 = model3.predict(X_test_scale)
# keep probabilities for the positive outcome only
yhat3 = yhat3[:, 0]
# calculate roc curves
fpr, tpr, thresholds3 = roc_curve(y_test_scale, yhat3)
# calculate the g-mean for each threshold
gmeans3 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans3)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds3[ix], gmeans3[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Churning')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()

Best Threshold=0.223874, G-Mean=0.775

y_pred_e3=model3.predict(X_test)
y_pred_e3 = (y_pred_e3 > thresholds3[ix])
y_pred_e3

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [ True]])

#Calculating the confusion matrix 
from sklearn.metrics import confusion_matrix
cm3=confusion_matrix(y_test_scale, y_pred_e3)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm3, 
                      group_names=labels,
                      categories=categories, 
                      cmap='Blues')

#Accuracy as per the classification report 
from sklearn import metrics
cr3=metrics.classification_report(y_test_scale,y_pred_e3)
print(cr3)

              precision    recall  f1-score   support

         0.0       0.92      0.83      0.87      1585
         1.0       0.52      0.73      0.61       415

    accuracy                           0.80      2000
   macro avg       0.72      0.78      0.74      2000
weighted avg       0.84      0.80      0.82      2000

• The Dropout technique helped the model reduce the loss function of both train and validation. The F1 score also seems to be fine, with a decrease in the False Negative rate.

Model 5

Random Search CV

backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

def create_model_v4(lr,batch_size):  
    np.random.seed(1337)
    model = Sequential()
    model.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
    model.add(Dropout(0.3))
    #model.add(Dense(128,activation='relu',kernel_initializer='he_uniform'))
    model.add(Dense(128,activation='relu'))
    model.add(Dropout(0.3))
    model.add(Dense(64,activation='relu'))
    model.add(Dropout(0.2))
    #model.add(Dense(32,activation='relu',kernel_initializer='he_uniform'))
    #model.add(Dropout(0.3))
    model.add(Dense(32,activation='relu')) 
    model.add(Dense(1, activation='sigmoid'))

    #compile model
    optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
    model.compile(optimizer = optimizer,loss = 'binary_crossentropy', metrics = ['accuracy'])
    return model

from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV

keras_estimator = KerasClassifier(build_fn=create_model_v4, verbose=1)
# define the grid search parameters
param_random = {
    'batch_size':[32, 64, 128],
    "lr":[0.01,0.1,0.001],}

kfold_splits = 3
random= RandomizedSearchCV(estimator=keras_estimator,  
                    verbose=1,
                    cv=kfold_splits,  
                    param_distributions=param_random,n_jobs=-1)

X_train_scale

<tf.Tensor: shape=(6400, 12), dtype=float32, numpy=
array([[0.70726776, 0.32799995, 0.22972974, ..., 0.        , 0.        ,
        1.        ],
       [0.5607605 , 0.804     , 0.24324325, ..., 0.        , 1.        ,
        1.        ],
       [0.7751503 , 0.58800006, 0.29729733, ..., 0.        , 0.        ,
        0.        ],
       ...,
       [0.55343246, 0.72      , 0.5945946 , ..., 0.        , 1.        ,
        1.        ],
       [0.86558914, 0.6700001 , 0.2027027 , ..., 0.        , 0.        ,
        1.        ],
       [0.42694092, 0.48800004, 0.47297296, ..., 0.        , 0.        ,
        0.        ]], dtype=float32)>

random_result = random.fit(X_train, y_train,validation_split=0.2,verbose=1) 

# Summarize results
print("Best: %f using %s" % (random_result.best_score_, random_result.best_params_))
means = random_result.cv_results_['mean_test_score']
stds = random_result.cv_results_['std_test_score']
params = random_result.cv_results_['params']

Fitting 3 folds for each of 9 candidates, totalling 27 fits
107/107 [==============================] - 2s 9ms/step - loss: 0.4883 - accuracy: 0.7981 - val_loss: 0.5066 - val_accuracy: 0.7775
107/107 [==============================] - 2s 9ms/step - loss: 0.4919 - accuracy: 0.8011 - val_loss: 0.4556 - val_accuracy: 0.8162
107/107 [==============================] - 2s 9ms/step - loss: 0.4975 - accuracy: 0.7940 - val_loss: 0.5071 - val_accuracy: 0.7775
107/107 [==============================] - 2s 9ms/step - loss: 1.1301 - accuracy: 0.7931 - val_loss: 0.5252 - val_accuracy: 0.7775
107/107 [==============================] - 2s 9ms/step - loss: 1.1195 - accuracy: 0.7902 - val_loss: 0.4778 - val_accuracy: 0.8162
107/107 [==============================] - 2s 8ms/step - loss: 0.7842 - accuracy: 0.7861 - val_loss: 0.5301 - val_accuracy: 0.7775
54/54 [==============================] - 2s 10ms/step - loss: 0.4867 - accuracy: 0.7931 - val_loss: 0.4951 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.4859 - accuracy: 0.8045 - val_loss: 0.4982 - val_accuracy: 0.7775
107/107 [==============================] - 2s 7ms/step - loss: 0.5045 - accuracy: 0.7887 - val_loss: 0.4941 - val_accuracy: 0.7775
54/54 [==============================] - 2s 9ms/step - loss: 0.5174 - accuracy: 0.7887 - val_loss: 0.4556 - val_accuracy: 0.8162
54/54 [==============================] - 2s 9ms/step - loss: 0.4950 - accuracy: 0.7882 - val_loss: 0.4865 - val_accuracy: 0.7775
107/107 [==============================] - 2s 7ms/step - loss: 0.4953 - accuracy: 0.7984 - val_loss: 0.4556 - val_accuracy: 0.8162
67/67 [==============================] - 0s 2ms/step - loss: 0.4735 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.4742 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.5020 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.4793 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4610 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.5260 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.4867 - accuracy: 0.8129
67/67 [==============================] - 0s 2ms/step - loss: 0.4646 - accuracy: 0.7957
34/34 [==============================] - 0s 2ms/step - loss: 0.4578 - accuracy: 0.8129
67/67 [==============================] - 0s 2ms/step - loss: 0.4591 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4748 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.4764 - accuracy: 0.7857
27/27 [==============================] - 2s 25ms/step - loss: 0.4855 - accuracy: 0.7937 - val_loss: 0.4856 - val_accuracy: 0.7775
54/54 [==============================] - 2s 14ms/step - loss: 1.9249 - accuracy: 0.7879 - val_loss: 0.4782 - val_accuracy: 0.8162
54/54 [==============================] - 2s 14ms/step - loss: 1.6557 - accuracy: 0.7975 - val_loss: 0.5376 - val_accuracy: 0.7775
27/27 [==============================] - 2s 22ms/step - loss: 0.4904 - accuracy: 0.7841 - val_loss: 0.5239 - val_accuracy: 0.7775
54/54 [==============================] - 2s 14ms/step - loss: 1.3663 - accuracy: 0.7823 - val_loss: 0.5327 - val_accuracy: 0.7775
27/27 [==============================] - 2s 18ms/step - loss: 2.6068 - accuracy: 0.7860 - val_loss: 0.5263 - val_accuracy: 0.7775
27/27 [==============================] - 2s 18ms/step - loss: 0.5068 - accuracy: 0.7879 - val_loss: 0.4697 - val_accuracy: 0.8162
54/54 [==============================] - 2s 13ms/step - loss: 0.5126 - accuracy: 0.7887 - val_loss: 0.5025 - val_accuracy: 0.7775
27/27 [==============================] - 2s 19ms/step - loss: 1.9176 - accuracy: 0.7627 - val_loss: 0.5328 - val_accuracy: 0.7775
54/54 [==============================] - 2s 13ms/step - loss: 0.5112 - accuracy: 0.7998 - val_loss: 0.5050 - val_accuracy: 0.7775
54/54 [==============================] - 2s 12ms/step - loss: 0.5365 - accuracy: 0.7770 - val_loss: 0.4601 - val_accuracy: 0.8162
27/27 [==============================] - 2s 14ms/step - loss: 2.7694 - accuracy: 0.7829 - val_loss: 0.4767 - val_accuracy: 0.8162
17/17 [==============================] - 0s 2ms/step - loss: 0.4639 - accuracy: 0.7957
17/17 [==============================] - 0s 2ms/step - loss: 0.4716 - accuracy: 0.8129
17/17 [==============================] - 0s 3ms/step - loss: 0.5000 - accuracy: 0.7957
17/17 [==============================] - 0s 2ms/step - loss: 0.4834 - accuracy: 0.7857
17/17 [==============================] - 0s 3ms/step - loss: 0.4820 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.5093 - accuracy: 0.7957
34/34 [==============================] - 0s 2ms/step - loss: 0.5200 - accuracy: 0.7857
34/34 [==============================] - 0s 3ms/step - loss: 0.4819 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4652 - accuracy: 0.8129
34/34 [==============================] - 0s 3ms/step - loss: 0.4835 - accuracy: 0.7957
34/34 [==============================] - 0s 3ms/step - loss: 0.4936 - accuracy: 0.7857
17/17 [==============================] - 0s 4ms/step - loss: 0.5241 - accuracy: 0.7857
27/27 [==============================] - 1s 10ms/step - loss: 0.5499 - accuracy: 0.7734 - val_loss: 0.5147 - val_accuracy: 0.7775
27/27 [==============================] - 1s 10ms/step - loss: 0.5519 - accuracy: 0.7653 - val_loss: 0.5118 - val_accuracy: 0.7775
27/27 [==============================] - 1s 9ms/step - loss: 0.5584 - accuracy: 0.7732 - val_loss: 0.4660 - val_accuracy: 0.8162
17/17 [==============================] - 0s 980us/step - loss: 0.4912 - accuracy: 0.7957
17/17 [==============================] - 0s 980us/step - loss: 0.4702 - accuracy: 0.8129
17/17 [==============================] - 0s 821us/step - loss: 0.5037 - accuracy: 0.7857
160/160 [==============================] - 1s 2ms/step - loss: 0.4794 - accuracy: 0.7982 - val_loss: 0.5614 - val_accuracy: 0.7906
Best: 0.798125 using {'lr': 0.01, 'batch_size': 32}

estimator_v4=create_model_v4(batch_size=random_result.best_params_['batch_size'],lr=random_result.best_params_['lr'])

estimator_v4.summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_5 (Dense)             (None, 256)               3328      
                                                                 
 dropout_3 (Dropout)         (None, 256)               0         
                                                                 
 dense_6 (Dense)             (None, 128)               32896     
                                                                 
 dropout_4 (Dropout)         (None, 128)               0         
                                                                 
 dense_7 (Dense)             (None, 64)                8256      
                                                                 
 dropout_5 (Dropout)         (None, 64)                0         
                                                                 
 dense_8 (Dense)             (None, 32)                2080      
                                                                 
 dense_9 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 46,593
Trainable params: 46,593
Non-trainable params: 0
_________________________________________________________________

optimizer = tf.keras.optimizers.Adam(random_result.best_params_['lr'])
estimator_v4.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])
history_4=estimator_v4.fit(X_train, y_train, epochs=50, batch_size = 64, verbose=1,validation_split=0.2)

Epoch 1/50
80/80 [==============================] - 1s 3ms/step - loss: 0.4759 - accuracy: 0.7986 - val_loss: 0.4963 - val_accuracy: 0.7906
Epoch 2/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4559 - accuracy: 0.8025 - val_loss: 0.4282 - val_accuracy: 0.8250
Epoch 3/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4251 - accuracy: 0.8180 - val_loss: 0.4176 - val_accuracy: 0.8398
Epoch 4/50
80/80 [==============================] - 0s 1ms/step - loss: 0.4090 - accuracy: 0.8268 - val_loss: 0.4047 - val_accuracy: 0.8328
Epoch 5/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3763 - accuracy: 0.8463 - val_loss: 0.3881 - val_accuracy: 0.8406
Epoch 6/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3883 - accuracy: 0.8328 - val_loss: 0.3742 - val_accuracy: 0.8461
Epoch 7/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3818 - accuracy: 0.8373 - val_loss: 0.3668 - val_accuracy: 0.8430
Epoch 8/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3766 - accuracy: 0.8436 - val_loss: 0.3773 - val_accuracy: 0.8508
Epoch 9/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3706 - accuracy: 0.8465 - val_loss: 0.3648 - val_accuracy: 0.8445
Epoch 10/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3719 - accuracy: 0.8473 - val_loss: 0.3741 - val_accuracy: 0.8531
Epoch 11/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3735 - accuracy: 0.8508 - val_loss: 0.3788 - val_accuracy: 0.8516
Epoch 12/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3685 - accuracy: 0.8432 - val_loss: 0.3832 - val_accuracy: 0.8516
Epoch 13/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3681 - accuracy: 0.8486 - val_loss: 0.3705 - val_accuracy: 0.8531
Epoch 14/50
80/80 [==============================] - 0s 1ms/step - loss: 0.3635 - accuracy: 0.8549 - val_loss: 0.3763 - val_accuracy: 0.8516
Epoch 15/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3609 - accuracy: 0.8518 - val_loss: 0.3628 - val_accuracy: 0.8570
Epoch 16/50
80/80 [==============================] - 0s 1ms/step - loss: 0.3591 - accuracy: 0.8553 - val_loss: 0.3775 - val_accuracy: 0.8500
Epoch 17/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3622 - accuracy: 0.8502 - val_loss: 0.3658 - val_accuracy: 0.8547
Epoch 18/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3616 - accuracy: 0.8562 - val_loss: 0.3664 - val_accuracy: 0.8547
Epoch 19/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3622 - accuracy: 0.8516 - val_loss: 0.3710 - val_accuracy: 0.8539
Epoch 20/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3579 - accuracy: 0.8537 - val_loss: 0.3619 - val_accuracy: 0.8453
Epoch 21/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3634 - accuracy: 0.8508 - val_loss: 0.3781 - val_accuracy: 0.8602
Epoch 22/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3612 - accuracy: 0.8529 - val_loss: 0.3656 - val_accuracy: 0.8547
Epoch 23/50
80/80 [==============================] - 0s 3ms/step - loss: 0.3569 - accuracy: 0.8545 - val_loss: 0.3692 - val_accuracy: 0.8531
Epoch 24/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3610 - accuracy: 0.8568 - val_loss: 0.3752 - val_accuracy: 0.8531
Epoch 25/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3610 - accuracy: 0.8512 - val_loss: 0.3632 - val_accuracy: 0.8469
Epoch 26/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3588 - accuracy: 0.8506 - val_loss: 0.3640 - val_accuracy: 0.8469
Epoch 27/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3621 - accuracy: 0.8518 - val_loss: 0.3699 - val_accuracy: 0.8547
Epoch 28/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3605 - accuracy: 0.8533 - val_loss: 0.3709 - val_accuracy: 0.8508
Epoch 29/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3598 - accuracy: 0.8523 - val_loss: 0.3677 - val_accuracy: 0.8539
Epoch 30/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3502 - accuracy: 0.8547 - val_loss: 0.3634 - val_accuracy: 0.8516
Epoch 31/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3530 - accuracy: 0.8504 - val_loss: 0.3743 - val_accuracy: 0.8562
Epoch 32/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3511 - accuracy: 0.8535 - val_loss: 0.3610 - val_accuracy: 0.8531
Epoch 33/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3483 - accuracy: 0.8541 - val_loss: 0.3994 - val_accuracy: 0.8562
Epoch 34/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3538 - accuracy: 0.8559 - val_loss: 0.3675 - val_accuracy: 0.8539
Epoch 35/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3602 - accuracy: 0.8541 - val_loss: 0.3685 - val_accuracy: 0.8516
Epoch 36/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3594 - accuracy: 0.8500 - val_loss: 0.3665 - val_accuracy: 0.8516
Epoch 37/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3517 - accuracy: 0.8574 - val_loss: 0.3620 - val_accuracy: 0.8555
Epoch 38/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3516 - accuracy: 0.8539 - val_loss: 0.3639 - val_accuracy: 0.8555
Epoch 39/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3500 - accuracy: 0.8531 - val_loss: 0.3656 - val_accuracy: 0.8461
Epoch 40/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3564 - accuracy: 0.8551 - val_loss: 0.3642 - val_accuracy: 0.8508
Epoch 41/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3491 - accuracy: 0.8551 - val_loss: 0.3615 - val_accuracy: 0.8547
Epoch 42/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3562 - accuracy: 0.8529 - val_loss: 0.3666 - val_accuracy: 0.8523
Epoch 43/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3562 - accuracy: 0.8516 - val_loss: 0.3716 - val_accuracy: 0.8508
Epoch 44/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3488 - accuracy: 0.8572 - val_loss: 0.3554 - val_accuracy: 0.8578
Epoch 45/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3584 - accuracy: 0.8562 - val_loss: 0.3741 - val_accuracy: 0.8516
Epoch 46/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3482 - accuracy: 0.8592 - val_loss: 0.3619 - val_accuracy: 0.8477
Epoch 47/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3555 - accuracy: 0.8527 - val_loss: 0.3627 - val_accuracy: 0.8469
Epoch 48/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3470 - accuracy: 0.8590 - val_loss: 0.3644 - val_accuracy: 0.8500
Epoch 49/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3484 - accuracy: 0.8578 - val_loss: 0.3621 - val_accuracy: 0.8484
Epoch 50/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3508 - accuracy: 0.8539 - val_loss: 0.3562 - val_accuracy: 0.8547

#Plotting Train Loss vs Validation Loss
plt.plot(history_4.history['loss'])
plt.plot(history_4.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

from sklearn.metrics import roc_curve

from matplotlib import pyplot


# predict probabilities
yhat4 = estimator_v4.predict(X_test)
# keep probabilities for the positive outcome only
yhat4 = yhat4[:, 0]
# calculate roc curves
fpr, tpr, thresholds4 = roc_curve(y_test, yhat4)
# calculate the g-mean for each threshold
gmeans4 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans4)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds4[ix], gmeans4[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Skill')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()

Best Threshold=0.204202, G-Mean=0.779

y_pred_e4=estimator_v4.predict(X_test)
y_pred_e4 = (y_pred_e4 > thresholds4[ix])
y_pred_e4

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [ True]])

#Calculating the confusion matrix 
from sklearn.metrics import confusion_matrix
cm4=confusion_matrix(y_test, y_pred_e4)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Changing Job','Changing Job']
make_confusion_matrix(cm4, 
                      group_names=labels,
                      categories=categories, 
                      cmap='Blues')

#Accuracy as per the classification report 
from sklearn import metrics
cr4=metrics.classification_report(y_test,y_pred_e4)
print(cr4)

              precision    recall  f1-score   support

         0.0       0.93      0.80      0.86      1585
         1.0       0.49      0.76      0.60       415

    accuracy                           0.79      2000
   macro avg       0.71      0.78      0.73      2000
weighted avg       0.84      0.79      0.80      2000

Accuracy, f1 score has decreased little however the ROC curve is smooth between train and validation

Model 6

Grid Search CV

backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

def create_model_v5(lr,batch_size):  
    np.random.seed(1337)
    model = Sequential()
    model.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
    model.add(Dropout(0.3))
    #model.add(Dense(128,activation='relu',kernel_initializer='he_uniform'))
    model.add(Dense(128,activation='relu'))
    model.add(Dropout(0.3))
    model.add(Dense(64,activation='relu'))
    model.add(Dropout(0.2))
    #model.add(Dense(32,activation='relu',kernel_initializer='he_uniform'))
    #model.add(Dropout(0.3))
    model.add(Dense(32,activation='relu')) 
    model.add(Dense(1, activation='sigmoid'))

    #compile model
    optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
    model.compile(optimizer = optimizer,loss = 'binary_crossentropy', metrics = ['accuracy'])
    return model

keras_estimator = KerasClassifier(build_fn=create_model_v5, verbose=1)
# define the grid search parameters
param_grid = {
    'batch_size':[64,32, 128],
    "lr":[0.01,0.1,0.001],}

kfold_splits = 3
grid = GridSearchCV(estimator=keras_estimator,  
                    verbose=1,
                    cv=kfold_splits,  
                    param_grid=param_grid,n_jobs=-1)

import time
  
# store starting time
begin = time.time()


grid_result = grid.fit(X_train, y_train,validation_split=0.2,verbose=1) 

# Summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']

time.sleep(1)
# store end time
end = time.time()
  
# total time taken
print(f"Total runtime of the program is {end - begin}")

Fitting 3 folds for each of 9 candidates, totalling 27 fits
54/54 [==============================] - 2s 15ms/step - loss: 1.8962 - accuracy: 0.7834 - val_loss: 0.5321 - val_accuracy: 0.7775
54/54 [==============================] - 2s 15ms/step - loss: 0.4962 - accuracy: 0.7926 - val_loss: 0.4664 - val_accuracy: 0.8162
54/54 [==============================] - 2s 16ms/step - loss: 0.4941 - accuracy: 0.7914 - val_loss: 0.4870 - val_accuracy: 0.7775
54/54 [==============================] - 2s 16ms/step - loss: 0.4827 - accuracy: 0.7943 - val_loss: 0.4894 - val_accuracy: 0.7775
54/54 [==============================] - 2s 15ms/step - loss: 1.1364 - accuracy: 0.7846 - val_loss: 0.5308 - val_accuracy: 0.7775
54/54 [==============================] - 2s 15ms/step - loss: 0.5069 - accuracy: 0.8007 - val_loss: 0.4977 - val_accuracy: 0.7775
54/54 [==============================] - 2s 15ms/step - loss: 0.5366 - accuracy: 0.7776 - val_loss: 0.4647 - val_accuracy: 0.8162
54/54 [==============================] - 2s 16ms/step - loss: 2.2594 - accuracy: 0.7984 - val_loss: 0.4787 - val_accuracy: 0.8162
54/54 [==============================] - 2s 15ms/step - loss: 0.5015 - accuracy: 0.7937 - val_loss: 0.5002 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.4833 - accuracy: 0.7923 - val_loss: 0.4773 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.4962 - accuracy: 0.8051 - val_loss: 0.4852 - val_accuracy: 0.7775
34/34 [==============================] - 0s 2ms/step - loss: 0.4583 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4831 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4956 - accuracy: 0.7857
34/34 [==============================] - 0s 2ms/step - loss: 0.4715 - accuracy: 0.7957
107/107 [==============================] - 2s 7ms/step - loss: 0.4842 - accuracy: 0.7958 - val_loss: 0.4701 - val_accuracy: 0.8162
34/34 [==============================] - 0s 4ms/step - loss: 0.5057 - accuracy: 0.7957
34/34 [==============================] - 0s 4ms/step - loss: 0.4819 - accuracy: 0.7857
34/34 [==============================] - 0s 4ms/step - loss: 0.5197 - accuracy: 0.7857
34/34 [==============================] - 0s 4ms/step - loss: 0.4575 - accuracy: 0.8129
34/34 [==============================] - 0s 5ms/step - loss: 0.4659 - accuracy: 0.7957
67/67 [==============================] - 0s 3ms/step - loss: 0.4631 - accuracy: 0.7957
67/67 [==============================] - 0s 3ms/step - loss: 0.4437 - accuracy: 0.8129
67/67 [==============================] - 0s 3ms/step - loss: 0.4804 - accuracy: 0.7857
27/27 [==============================] - 2s 29ms/step - loss: 0.4866 - accuracy: 0.8025 - val_loss: 0.4877 - val_accuracy: 0.7775
27/27 [==============================] - 2s 28ms/step - loss: 0.5073 - accuracy: 0.7814 - val_loss: 0.4748 - val_accuracy: 0.8162
27/27 [==============================] - 2s 26ms/step - loss: 0.5063 - accuracy: 0.7770 - val_loss: 0.4929 - val_accuracy: 0.7775
27/27 [==============================] - 2s 16ms/step - loss: 3.7837 - accuracy: 0.7987 - val_loss: 0.5368 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.7278 - accuracy: 0.7914 - val_loss: 0.4777 - val_accuracy: 0.8162
107/107 [==============================] - 2s 9ms/step - loss: 0.9144 - accuracy: 0.7966 - val_loss: 0.5311 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.9764 - accuracy: 0.7882 - val_loss: 0.5300 - val_accuracy: 0.7775
17/17 [==============================] - 0s 3ms/step - loss: 0.4671 - accuracy: 0.7957
17/17 [==============================] - 0s 3ms/step - loss: 0.4909 - accuracy: 0.7857
107/107 [==============================] - 2s 9ms/step - loss: 0.5251 - accuracy: 0.7668 - val_loss: 0.4966 - val_accuracy: 0.7775
27/27 [==============================] - 2s 17ms/step - loss: 2.8845 - accuracy: 0.7644 - val_loss: 0.5305 - val_accuracy: 0.7775
17/17 [==============================] - 0s 2ms/step - loss: 0.4579 - accuracy: 0.8129
107/107 [==============================] - 2s 8ms/step - loss: 0.5071 - accuracy: 0.7863 - val_loss: 0.4954 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.5058 - accuracy: 0.7908 - val_loss: 0.4599 - val_accuracy: 0.8162
27/27 [==============================] - 2s 16ms/step - loss: 3.4141 - accuracy: 0.7923 - val_loss: 0.4773 - val_accuracy: 0.8162
17/17 [==============================] - 0s 3ms/step - loss: 0.4927 - accuracy: 0.7957
17/17 [==============================] - 0s 3ms/step - loss: 0.4836 - accuracy: 0.8129
67/67 [==============================] - 0s 2ms/step - loss: 0.5205 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.5063 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.4854 - accuracy: 0.8129
67/67 [==============================] - 0s 2ms/step - loss: 0.4582 - accuracy: 0.8129
17/17 [==============================] - 0s 2ms/step - loss: 0.5212 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.4685 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.4922 - accuracy: 0.7857
27/27 [==============================] - 1s 8ms/step - loss: 0.5340 - accuracy: 0.7870 - val_loss: 0.4781 - val_accuracy: 0.8162
27/27 [==============================] - 1s 8ms/step - loss: 0.5269 - accuracy: 0.7805 - val_loss: 0.5099 - val_accuracy: 0.7775
27/27 [==============================] - 1s 8ms/step - loss: 0.5165 - accuracy: 0.7940 - val_loss: 0.5054 - val_accuracy: 0.7775
17/17 [==============================] - 0s 1ms/step - loss: 0.5029 - accuracy: 0.7857
17/17 [==============================] - 0s 1ms/step - loss: 0.4845 - accuracy: 0.7957
17/17 [==============================] - 0s 999us/step - loss: 0.4709 - accuracy: 0.8129
80/80 [==============================] - 1s 3ms/step - loss: 0.4741 - accuracy: 0.7969 - val_loss: 0.4620 - val_accuracy: 0.7906
Best: 0.798125 using {'batch_size': 64, 'lr': 0.01}
Total runtime of the program is 7.652716636657715

estimator_v5=create_model_v5(batch_size=grid_result.best_params_['batch_size'],lr=grid_result.best_params_['lr'])

estimator_v5.summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_5 (Dense)             (None, 256)               3328      
                                                                 
 dropout_3 (Dropout)         (None, 256)               0         
                                                                 
 dense_6 (Dense)             (None, 128)               32896     
                                                                 
 dropout_4 (Dropout)         (None, 128)               0         
                                                                 
 dense_7 (Dense)             (None, 64)                8256      
                                                                 
 dropout_5 (Dropout)         (None, 64)                0         
                                                                 
 dense_8 (Dense)             (None, 32)                2080      
                                                                 
 dense_9 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 46,593
Trainable params: 46,593
Non-trainable params: 0
_________________________________________________________________

optimizer = tf.keras.optimizers.Adam(grid_result.best_params_['lr'])
estimator_v5.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])
history_5=estimator_v5.fit(X_train, y_train, epochs=50, batch_size = 64, verbose=1,validation_split=0.2)

Epoch 1/50
80/80 [==============================] - 1s 3ms/step - loss: 0.4759 - accuracy: 0.7986 - val_loss: 0.4963 - val_accuracy: 0.7906
Epoch 2/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4559 - accuracy: 0.8025 - val_loss: 0.4282 - val_accuracy: 0.8250
Epoch 3/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4251 - accuracy: 0.8180 - val_loss: 0.4176 - val_accuracy: 0.8398
Epoch 4/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4090 - accuracy: 0.8268 - val_loss: 0.4047 - val_accuracy: 0.8328
Epoch 5/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3763 - accuracy: 0.8463 - val_loss: 0.3881 - val_accuracy: 0.8406
Epoch 6/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3883 - accuracy: 0.8328 - val_loss: 0.3742 - val_accuracy: 0.8461
Epoch 7/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3818 - accuracy: 0.8373 - val_loss: 0.3668 - val_accuracy: 0.8430
Epoch 8/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3766 - accuracy: 0.8436 - val_loss: 0.3773 - val_accuracy: 0.8508
Epoch 9/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3706 - accuracy: 0.8465 - val_loss: 0.3648 - val_accuracy: 0.8445
Epoch 10/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3719 - accuracy: 0.8473 - val_loss: 0.3741 - val_accuracy: 0.8531
Epoch 11/50
80/80 [==============================] - 0s 3ms/step - loss: 0.3735 - accuracy: 0.8508 - val_loss: 0.3788 - val_accuracy: 0.8516
Epoch 12/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3685 - accuracy: 0.8432 - val_loss: 0.3832 - val_accuracy: 0.8516
Epoch 13/50
80/80 [==============================] - 0s 1ms/step - loss: 0.3681 - accuracy: 0.8486 - val_loss: 0.3705 - val_accuracy: 0.8531
Epoch 14/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3635 - accuracy: 0.8549 - val_loss: 0.3763 - val_accuracy: 0.8516
Epoch 15/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3609 - accuracy: 0.8518 - val_loss: 0.3628 - val_accuracy: 0.8570
Epoch 16/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3591 - accuracy: 0.8553 - val_loss: 0.3775 - val_accuracy: 0.8500
Epoch 17/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3622 - accuracy: 0.8502 - val_loss: 0.3658 - val_accuracy: 0.8547
Epoch 18/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3616 - accuracy: 0.8562 - val_loss: 0.3664 - val_accuracy: 0.8547
Epoch 19/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3622 - accuracy: 0.8516 - val_loss: 0.3710 - val_accuracy: 0.8539
Epoch 20/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3579 - accuracy: 0.8537 - val_loss: 0.3619 - val_accuracy: 0.8453
Epoch 21/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3634 - accuracy: 0.8508 - val_loss: 0.3781 - val_accuracy: 0.8602
Epoch 22/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3612 - accuracy: 0.8529 - val_loss: 0.3656 - val_accuracy: 0.8547
Epoch 23/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3569 - accuracy: 0.8545 - val_loss: 0.3692 - val_accuracy: 0.8531
Epoch 24/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3610 - accuracy: 0.8568 - val_loss: 0.3752 - val_accuracy: 0.8531
Epoch 25/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3610 - accuracy: 0.8512 - val_loss: 0.3632 - val_accuracy: 0.8469
Epoch 26/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3588 - accuracy: 0.8506 - val_loss: 0.3640 - val_accuracy: 0.8469
Epoch 27/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3621 - accuracy: 0.8518 - val_loss: 0.3699 - val_accuracy: 0.8547
Epoch 28/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3605 - accuracy: 0.8533 - val_loss: 0.3709 - val_accuracy: 0.8508
Epoch 29/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3598 - accuracy: 0.8523 - val_loss: 0.3677 - val_accuracy: 0.8539
Epoch 30/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3502 - accuracy: 0.8547 - val_loss: 0.3634 - val_accuracy: 0.8516
Epoch 31/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3530 - accuracy: 0.8504 - val_loss: 0.3743 - val_accuracy: 0.8562
Epoch 32/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3511 - accuracy: 0.8535 - val_loss: 0.3610 - val_accuracy: 0.8531
Epoch 33/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3483 - accuracy: 0.8541 - val_loss: 0.3994 - val_accuracy: 0.8562
Epoch 34/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3538 - accuracy: 0.8559 - val_loss: 0.3675 - val_accuracy: 0.8539
Epoch 35/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3602 - accuracy: 0.8541 - val_loss: 0.3685 - val_accuracy: 0.8516
Epoch 36/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3594 - accuracy: 0.8500 - val_loss: 0.3665 - val_accuracy: 0.8516
Epoch 37/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3517 - accuracy: 0.8574 - val_loss: 0.3620 - val_accuracy: 0.8555
Epoch 38/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3516 - accuracy: 0.8539 - val_loss: 0.3639 - val_accuracy: 0.8555
Epoch 39/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3500 - accuracy: 0.8531 - val_loss: 0.3656 - val_accuracy: 0.8461
Epoch 40/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3564 - accuracy: 0.8551 - val_loss: 0.3642 - val_accuracy: 0.8508
Epoch 41/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3491 - accuracy: 0.8551 - val_loss: 0.3615 - val_accuracy: 0.8547
Epoch 42/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3562 - accuracy: 0.8529 - val_loss: 0.3666 - val_accuracy: 0.8523
Epoch 43/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3562 - accuracy: 0.8516 - val_loss: 0.3716 - val_accuracy: 0.8508
Epoch 44/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3488 - accuracy: 0.8572 - val_loss: 0.3554 - val_accuracy: 0.8578
Epoch 45/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3584 - accuracy: 0.8562 - val_loss: 0.3741 - val_accuracy: 0.8516
Epoch 46/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3482 - accuracy: 0.8592 - val_loss: 0.3619 - val_accuracy: 0.8477
Epoch 47/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3555 - accuracy: 0.8527 - val_loss: 0.3627 - val_accuracy: 0.8469
Epoch 48/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3470 - accuracy: 0.8590 - val_loss: 0.3644 - val_accuracy: 0.8500
Epoch 49/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3484 - accuracy: 0.8578 - val_loss: 0.3621 - val_accuracy: 0.8484
Epoch 50/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3508 - accuracy: 0.8539 - val_loss: 0.3562 - val_accuracy: 0.8547

#Plotting Train Loss vs Validation Loss
plt.plot(history_5.history['loss'])
plt.plot(history_5.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

• ROC curve is smooth

from sklearn.metrics import roc_curve

from matplotlib import pyplot


# predict probabilities
yhat5 = estimator_v5.predict(X_test)
# keep probabilities for the positive outcome only
yhat5 = yhat5[:, 0]
# calculate roc curves
fpr, tpr, thresholds5 = roc_curve(y_test, yhat5)
# calculate the g-mean for each threshold
gmeans5 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans5)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds5[ix], gmeans5[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Churn')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()

Best Threshold=0.204202, G-Mean=0.779

y_pred_e5=estimator_v5.predict(X_test)
y_pred_e5 = (y_pred_e5 > thresholds5[ix])
y_pred_e5

array([[False],
       [False],
       [False],
       ...,
       [False],
       [False],
       [ True]])

#Calculating the confusion matrix 
from sklearn.metrics import confusion_matrix
cm5=confusion_matrix(y_test, y_pred_e5)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm5, 
                      group_names=labels,
                      categories=categories, 
                      cmap='Blues')

#Accuracy as per the classification report 
from sklearn import metrics
cr5=metrics.classification_report(y_test,y_pred_e5)
print(cr5)

              precision    recall  f1-score   support

         0.0       0.93      0.80      0.86      1585
         1.0       0.49      0.76      0.60       415

    accuracy                           0.79      2000
   macro avg       0.71      0.78      0.73      2000
weighted avg       0.84      0.79      0.80      2000

from tensorflow import keras
from tensorflow.keras import layers
from keras_tuner.tuners import RandomSearch

backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

def build_model(h):
    model = keras.Sequential()
    for i in range(h.Int('num_layers', 2, 10)):
        model.add(layers.Dense(units=h.Int('units_' + str(i),
                                            min_value=32,
                                            max_value=256,
                                            step=32),
                               activation='relu'))
    model.add(layers.Dense(1, activation='sigmoid'))
    model.compile(
        optimizer=keras.optimizers.Adam(
            h.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),
        loss='binary_crossentropy',
        metrics=['accuracy'])
    return model

tuner = RandomSearch(
    build_model,
    objective='val_accuracy',
    max_trials=5,
    executions_per_trial=3,
     project_name='Job_')

tuner.search_space_summary()

Search space summary
Default search space size: 4
num_layers (Int)
{'default': None, 'conditions': [], 'min_value': 2, 'max_value': 10, 'step': 1, 'sampling': None}
units_0 (Int)
{'default': None, 'conditions': [], 'min_value': 32, 'max_value': 256, 'step': 32, 'sampling': None}
units_1 (Int)
{'default': None, 'conditions': [], 'min_value': 32, 'max_value': 256, 'step': 32, 'sampling': None}
learning_rate (Choice)
{'default': 0.01, 'conditions': [], 'values': [0.01, 0.001, 0.0001], 'ordered': True}

### Searching the best model on X and y train
tuner.search(X_train, y_train,
             epochs=5,
             validation_split = 0.2)

Trial 5 Complete [00h 00m 07s]
val_accuracy: 0.8158854047457377

Best val_accuracy So Far: 0.8565104206403097
Total elapsed time: 00h 00m 33s
INFO:tensorflow:Oracle triggered exit

## Printing the best models with their hyperparameters
tuner.results_summary()

Results summary
Results in ./Job_
Showing 10 best trials
<keras_tuner.engine.objective.Objective object at 0x7fdb19aa7ac0>
Trial summary
Hyperparameters:
num_layers: 5
units_0: 64
units_1: 96
learning_rate: 0.001
units_2: 96
units_3: 224
units_4: 256
units_5: 160
units_6: 192
units_7: 224
units_8: 224
Score: 0.8565104206403097
Trial summary
Hyperparameters:
num_layers: 5
units_0: 160
units_1: 160
learning_rate: 0.001
units_2: 224
units_3: 128
units_4: 224
units_5: 64
units_6: 160
units_7: 64
units_8: 32
Score: 0.854687511920929
Trial summary
Hyperparameters:
num_layers: 8
units_0: 160
units_1: 64
learning_rate: 0.001
units_2: 160
units_3: 64
units_4: 192
units_5: 32
units_6: 224
units_7: 96
units_8: 192
Score: 0.854687492052714
Trial summary
Hyperparameters:
num_layers: 9
units_0: 128
units_1: 32
learning_rate: 0.0001
units_2: 160
units_3: 160
units_4: 160
units_5: 192
units_6: 96
units_7: 128
units_8: 96
Score: 0.8158854047457377
Trial summary
Hyperparameters:
num_layers: 9
units_0: 192
units_1: 96
learning_rate: 0.0001
units_2: 32
units_3: 32
units_4: 32
units_5: 32
units_6: 32
units_7: 32
units_8: 32
Score: 0.7911458214124044

0.8565104206403097 best accuracy so far with 5 layers

Model - Keras Tuner based recommendation

backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

model7 = Sequential()
model7.add(Dense(64,activation='relu',kernel_initializer='he_uniform',input_dim = X_train.shape[1]))
model7.add(Dense(96,activation='relu',kernel_initializer='he_uniform'))
model7.add(Dense(96,activation='relu',kernel_initializer='he_uniform'))
model7.add(Dense(224,activation='relu',kernel_initializer='he_uniform'))
model7.add(Dense(256,activation='relu',kernel_initializer='he_uniform'))
model7.add(Dense(1, activation = 'sigmoid'))

model7.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 64)                832       
                                                                 
 dense_1 (Dense)             (None, 96)                6240      
                                                                 
 dense_2 (Dense)             (None, 96)                9312      
                                                                 
 dense_3 (Dense)             (None, 224)               21728     
                                                                 
 dense_4 (Dense)             (None, 256)               57600     
                                                                 
 dense_5 (Dense)             (None, 1)                 257       
                                                                 
=================================================================
Total params: 95,969
Trainable params: 95,969
Non-trainable params: 0
_________________________________________________________________

optimizer = tf.keras.optimizers.Adam(0.001)
model7.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])

history_7 = model7.fit(X_train,y_train,batch_size=64,epochs=50,verbose=1,validation_split = 0.2)

Epoch 1/50
80/80 [==============================] - 1s 3ms/step - loss: 0.4649 - accuracy: 0.8021 - val_loss: 0.4701 - val_accuracy: 0.8062
Epoch 2/50
80/80 [==============================] - 0s 2ms/step - loss: 0.4301 - accuracy: 0.8156 - val_loss: 0.4187 - val_accuracy: 0.8227
Epoch 3/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3922 - accuracy: 0.8346 - val_loss: 0.4038 - val_accuracy: 0.8414
Epoch 4/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3642 - accuracy: 0.8453 - val_loss: 0.3926 - val_accuracy: 0.8430
Epoch 5/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3474 - accuracy: 0.8553 - val_loss: 0.3756 - val_accuracy: 0.8484
Epoch 6/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3462 - accuracy: 0.8543 - val_loss: 0.3914 - val_accuracy: 0.8336
Epoch 7/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3296 - accuracy: 0.8641 - val_loss: 0.3877 - val_accuracy: 0.8375
Epoch 8/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3263 - accuracy: 0.8641 - val_loss: 0.3891 - val_accuracy: 0.8352
Epoch 9/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3192 - accuracy: 0.8668 - val_loss: 0.3833 - val_accuracy: 0.8352
Epoch 10/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3104 - accuracy: 0.8711 - val_loss: 0.4173 - val_accuracy: 0.8344
Epoch 11/50
80/80 [==============================] - 0s 2ms/step - loss: 0.3063 - accuracy: 0.8687 - val_loss: 0.3938 - val_accuracy: 0.8508
Epoch 12/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2974 - accuracy: 0.8701 - val_loss: 0.3895 - val_accuracy: 0.8367
Epoch 13/50
80/80 [==============================] - 0s 3ms/step - loss: 0.2927 - accuracy: 0.8797 - val_loss: 0.3951 - val_accuracy: 0.8367
Epoch 14/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2877 - accuracy: 0.8803 - val_loss: 0.4095 - val_accuracy: 0.8383
Epoch 15/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2748 - accuracy: 0.8846 - val_loss: 0.4051 - val_accuracy: 0.8359
Epoch 16/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2711 - accuracy: 0.8844 - val_loss: 0.4207 - val_accuracy: 0.8195
Epoch 17/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2687 - accuracy: 0.8844 - val_loss: 0.4224 - val_accuracy: 0.8258
Epoch 18/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2587 - accuracy: 0.8953 - val_loss: 0.4372 - val_accuracy: 0.8492
Epoch 19/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2552 - accuracy: 0.8910 - val_loss: 0.4543 - val_accuracy: 0.8422
Epoch 20/50
80/80 [==============================] - 0s 3ms/step - loss: 0.2479 - accuracy: 0.8934 - val_loss: 0.4422 - val_accuracy: 0.8438
Epoch 21/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2403 - accuracy: 0.8990 - val_loss: 0.4658 - val_accuracy: 0.8281
Epoch 22/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2306 - accuracy: 0.9021 - val_loss: 0.4546 - val_accuracy: 0.8336
Epoch 23/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2197 - accuracy: 0.9059 - val_loss: 0.4966 - val_accuracy: 0.8352
Epoch 24/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2194 - accuracy: 0.9029 - val_loss: 0.5221 - val_accuracy: 0.8406
Epoch 25/50
80/80 [==============================] - 0s 2ms/step - loss: 0.2094 - accuracy: 0.9113 - val_loss: 0.5080 - val_accuracy: 0.8070
Epoch 26/50
80/80 [==============================] - 0s 3ms/step - loss: 0.2048 - accuracy: 0.9094 - val_loss: 0.5256 - val_accuracy: 0.8203
Epoch 27/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1950 - accuracy: 0.9176 - val_loss: 0.5459 - val_accuracy: 0.8117
Epoch 28/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1863 - accuracy: 0.9195 - val_loss: 0.5628 - val_accuracy: 0.8242
Epoch 29/50
80/80 [==============================] - 0s 3ms/step - loss: 0.1698 - accuracy: 0.9283 - val_loss: 0.5982 - val_accuracy: 0.8180
Epoch 30/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1872 - accuracy: 0.9238 - val_loss: 0.5737 - val_accuracy: 0.8305
Epoch 31/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1754 - accuracy: 0.9252 - val_loss: 0.6082 - val_accuracy: 0.8297
Epoch 32/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1580 - accuracy: 0.9318 - val_loss: 0.6193 - val_accuracy: 0.8289
Epoch 33/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1437 - accuracy: 0.9377 - val_loss: 0.7511 - val_accuracy: 0.8289
Epoch 34/50
80/80 [==============================] - 0s 3ms/step - loss: 0.1463 - accuracy: 0.9373 - val_loss: 0.7235 - val_accuracy: 0.8336
Epoch 35/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1563 - accuracy: 0.9354 - val_loss: 0.6715 - val_accuracy: 0.7883
Epoch 36/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1509 - accuracy: 0.9381 - val_loss: 0.7063 - val_accuracy: 0.8047
Epoch 37/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1446 - accuracy: 0.9395 - val_loss: 0.7154 - val_accuracy: 0.8320
Epoch 38/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1438 - accuracy: 0.9387 - val_loss: 0.7652 - val_accuracy: 0.8211
Epoch 39/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1223 - accuracy: 0.9488 - val_loss: 0.7570 - val_accuracy: 0.8180
Epoch 40/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1124 - accuracy: 0.9531 - val_loss: 0.7971 - val_accuracy: 0.8211
Epoch 41/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1098 - accuracy: 0.9543 - val_loss: 0.9006 - val_accuracy: 0.8258
Epoch 42/50
80/80 [==============================] - 0s 3ms/step - loss: 0.1050 - accuracy: 0.9594 - val_loss: 0.8420 - val_accuracy: 0.8188
Epoch 43/50
80/80 [==============================] - 0s 2ms/step - loss: 0.0929 - accuracy: 0.9625 - val_loss: 0.9249 - val_accuracy: 0.8070
Epoch 44/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1017 - accuracy: 0.9604 - val_loss: 0.9175 - val_accuracy: 0.8016
Epoch 45/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1107 - accuracy: 0.9572 - val_loss: 0.8992 - val_accuracy: 0.8141
Epoch 46/50
80/80 [==============================] - 0s 2ms/step - loss: 0.1159 - accuracy: 0.9535 - val_loss: 0.9143 - val_accuracy: 0.8133
Epoch 47/50
80/80 [==============================] - 0s 2ms/step - loss: 0.0827 - accuracy: 0.9662 - val_loss: 0.9381 - val_accuracy: 0.8141
Epoch 48/50
80/80 [==============================] - 0s 2ms/step - loss: 0.0787 - accuracy: 0.9695 - val_loss: 1.0746 - val_accuracy: 0.8062
Epoch 49/50
80/80 [==============================] - 0s 2ms/step - loss: 0.0852 - accuracy: 0.9680 - val_loss: 1.0367 - val_accuracy: 0.8172
Epoch 50/50
80/80 [==============================] - 0s 2ms/step - loss: 0.0779 - accuracy: 0.9695 - val_loss: 1.0493 - val_accuracy: 0.8117

#Plotting Train Loss vs Validation Loss
plt.plot(history_7.history['loss'])
plt.plot(history_7.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

• ROC curve isn't ideal between train and validation

from sklearn.metrics import roc_curve

from matplotlib import pyplot


# predict probabilities
yhat7 = model7.predict(X_test)
# keep probabilities for the positive outcome only
yhat7 = yhat7[:, 0]
# calculate roc curves
fpr, tpr, thresholds7 = roc_curve(y_test, yhat7)
# calculate the g-mean for each threshold
gmeans7 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans7)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds7[ix], gmeans7[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Skill')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()

Best Threshold=0.023485, G-Mean=0.717

y_pred_e7=model7.predict(X_test)
y_pred_e7 = (y_pred_e7 > thresholds7[ix])
y_pred_e7

array([[False],
       [ True],
       [False],
       ...,
       [False],
       [False],
       [ True]])

#Calculating the confusion matrix 
from sklearn.metrics import confusion_matrix
cm7=confusion_matrix(y_test, y_pred_e7)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm7, 
                      group_names=labels,
                      categories=categories, 
                      cmap='Blues')

#Accuracy as per the classification report 
from sklearn import metrics
cr7=metrics.classification_report(y_test,y_pred_e7)
print(cr7)

              precision    recall  f1-score   support

         0.0       0.91      0.71      0.80      1585
         1.0       0.39      0.72      0.51       415

    accuracy                           0.71      2000
   macro avg       0.65      0.72      0.65      2000
weighted avg       0.80      0.71      0.74      2000

Model Performance Evaluation

• Evaluate the model on different performance metrics and comment on the performance and scope of improvement

print(cr3) # with dropout
print(cr4)# Random Search CV
print(cr5)# Grid Search CV
print(cr7) #Keras Tuner

              precision    recall  f1-score   support

         0.0       0.92      0.83      0.87      1585
         1.0       0.52      0.73      0.61       415

    accuracy                           0.80      2000
   macro avg       0.72      0.78      0.74      2000
weighted avg       0.84      0.80      0.82      2000

              precision    recall  f1-score   support

         0.0       0.93      0.80      0.86      1585
         1.0       0.49      0.76      0.60       415

    accuracy                           0.79      2000
   macro avg       0.71      0.78      0.73      2000
weighted avg       0.84      0.79      0.80      2000

              precision    recall  f1-score   support

         0.0       0.93      0.80      0.86      1585
         1.0       0.49      0.76      0.60       415

    accuracy                           0.79      2000
   macro avg       0.71      0.78      0.73      2000
weighted avg       0.84      0.79      0.80      2000

              precision    recall  f1-score   support

         0.0       0.91      0.71      0.80      1585
         1.0       0.39      0.72      0.51       415

    accuracy                           0.71      2000
   macro avg       0.65      0.72      0.65      2000
weighted avg       0.80      0.71      0.74      2000

Optimal Model

• Our model here can be Model 4 which uses the Dropout regularization technique and works on the imbalanced dataset.

• GridSearchCV, RandomSearch CV has better ROC curves model's F1 score, while better than in Randomized Search, is slightly lower than in Model 4 (the Dropout model) which can also be used.

• Any one Machine Learning model above can be used.

• We cab do better feature engineerning by removing the flaws of the skewed variables if required.

Conclusion and key takeaways

• The optimised model gives better results as seen from the confusion matrix as well as an improved F1 score. The dropout model 4 gives good options.
• The Bank can deploy the final model from this exercise to identify with a reasonable degree of accuracy to predict churn or not based.

• Based on the EDA, we can target the churning categories like older age group and based on other demography like country Germany etc as well. The model will learn better as data gets less imbalanced. Existing model gives excellent Acurracy and F1 score

• We encourage you to further look to optimize the model and come up with better results.

Thank you - Amogh